{"thread":{"id":"47943","subject":"Reduce pack-objects memory footprint?","startedAt":"2018-02-28T09:27:36Z","lastAt":"2018-04-06T21:47:45Z","messageCount":273,"participants":["Duy Nguyen","Jeff King","Eric Wong","Nguyễn Thái Ngọc Duy","Ævar Arnfjörð Bjarmason","Junio C Hamano","Johannes Schindelin","Ramsay Jones","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"340546","messageId":"20180228092722.GA25627@ash","threadId":"47943","inReplyTo":null,"subject":"Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-02-28T09:27:22Z","receivedAt":"2018-02-28T09:27:36Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\nconsumes 1.7G out of 4G RAM, pushing lots of data to swap and making\nall apps nearly unusuable (granted the problem is partly Linux I/O\nscheduler too). So I wonder if we can reduce pack-objects memory\nfootprint a bit.\n\nThis demonstration patch (probably breaks some tests) would reduce the\nsize of struct object_entry from from 136 down to 112 bytes on\nx86-64. There are 6483999 of these objects, so the saving is 17% or\n148 MB.\n\nIf we go further, notice that nr_objects is uint32_t, we could convert\nthe three pointers\n\n\tstruct object_entry *delta;\n\tstruct object_entry *delta_child;\n\tstruct object_entry *delta_sibling;\n\nto\n\n\tuint32_t delta;\n\tuint32_t delta_child;\n\tuint32_t delta_sibling;\n\nwhich saves 12 bytes (or another 74 MB). 222 MB total is plenty of\nspace to keep some file cache from being evicted.\n\nIs it worth doing this? The struct packing makes it harder to read\n(and more fragile too). I added some more artifical limit like max\ndepth of 2^11. But I think 4096+ depth is getting unreasonable.\n\n-- 8< --\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..6a9804daec 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -877,8 +877,11 @@ static void write_pack_file(void)\n \t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n \n \t\t\tif (write_bitmap_index) {\n+\t\t\t\tALLOC_ARRAY(to_pack.in_pack_pos, to_pack.nr_objects);\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(written_list,\n+\t\t\t\t\t\t\t nr_written,\n+\t\t\t\t\t\t\t &to_pack);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n@@ -1407,6 +1410,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,8 +1419,9 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n+\t\tentry->in_pack_type = type;\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n@@ -1580,7 +1586,8 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t */\n \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n \t\t\t\t\t &entry->size);\n-\t}\n+\t} else\n+\t\tentry->type = type;\n }\n \n /*\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..5c9957a095 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -49,7 +49,8 @@ void bitmap_writer_show_progress(int show)\n * Build the initial type index for the packfile\n */\n void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t struct packing_data *to_pack)\n {\n \tuint32_t i;\n \n@@ -62,7 +63,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\tto_pack->in_pack_pos[entry - to_pack->objects] = i;\n \n \t\tswitch (entry->type) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +148,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn writer.to_pack->in_pack_pos[entry - writer.to_pack->objects];\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..595914fa43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = mapping->in_pack_pos[oe - mapping->objects] + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..2558f7662a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t struct packing_data *);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..caecad23b6 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,18 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE\n+};\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n@@ -14,11 +26,10 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n+\tunsigned type:3;\t /* enum object_type */\n+\tunsigned in_pack_type:3; /* enum object_type - could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n@@ -27,19 +38,8 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n-\n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n-\tint depth;\n+\tunsigned dfs_state:3;\t/* enum dfs_state */\n+\tunsigned depth:11;\n };\n \n struct packing_data {\n@@ -48,6 +48,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tint *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n-- 8< --\n"},{"id":"340549","messageId":"20180228101757.GA11803@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-28T10:17:57Z","receivedAt":"2018-02-28T10:18:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 28, 2018 at 04:27:22PM +0700, Duy Nguyen wrote:\n\n> linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n> all apps nearly unusuable (granted the problem is partly Linux I/O\n> scheduler too). So I wonder if we can reduce pack-objects memory\n> footprint a bit.\n\nYeah, the per object memory footprint is not great. Around 100 million\nobjects it becomes pretty ridiculous. I started to dig into it a year or\nthree ago when I saw such a case, but it turned out to be something that\nwe could prune. The torvalds/linux fork network has ~23 million objects,\nso it's probably 7-8 GB of book-keeping. Which is gross, but 64GB in a\nserver isn't uncommon these days.\n\nI think laptops repacking the kernel are probably one of the worst cases\n(leaving aside the awful Windows repository, but my impression is that\nthey simply can't do a full repack at all there).\n\n> This demonstration patch (probably breaks some tests) would reduce the\n> size of struct object_entry from from 136 down to 112 bytes on\n> x86-64. There are 6483999 of these objects, so the saving is 17% or\n> 148 MB.\n\n136 x 6.5M objects is only about 800MB. I suspect a big chunk of the\nrest is going to the object structs we create when doing the internal\nrev-list traversal. And those duplicate the 20-byte sha1s at the very\nleast.\n\nI don't know if it would be a good idea to free them after the\ntraversal, though. We do use them again later in the bitmap case. On the\nother hand, we could probably do so for the non-bitmap case. And even\nfor the bitmap case, the primary value in keeping them around is that\nthe parent pointers will already be cached. So it might make sense to\nfree the blobs and trees (though it might be tricky; the commits have\npointers to the trees).\n\nIt also doesn't help with peak memory usage, because you'll have the\nfull to_pack list and all of the \"struct object\" in memory together at\none point.\n\nAnother option would be to somehow replace the pack_idx_entry with a\nreference to a \"struct object\". That would let us at least avoid storing\nthe 20-byte oid twice.\n\n> If we go further, notice that nr_objects is uint32_t, we could convert\n> the three pointers\n> \n> \tstruct object_entry *delta;\n> \tstruct object_entry *delta_child;\n> \tstruct object_entry *delta_sibling;\n> \n> to\n> \n> \tuint32_t delta;\n> \tuint32_t delta_child;\n> \tuint32_t delta_sibling;\n> \n> which saves 12 bytes (or another 74 MB). 222 MB total is plenty of\n> space to keep some file cache from being evicted.\n\nYeah, that seems like low-hanging fruit. I'd also note that we don't\nactually use all of the fields during the whole process. I think some of\nthose delta fields are only used for a short time. So we might be able\nto reduce peak memory if there are some mutually exclusive bits of each\nentry (and even if there's some overlap, we'd reduce the length of time\nwe'd need to be at peak).\n\n> Is it worth doing this? The struct packing makes it harder to read\n> (and more fragile too). I added some more artifical limit like max\n> depth of 2^11. But I think 4096+ depth is getting unreasonable.\n\nI'm OK with a limit like 4096, as long as we notice when we hit the\nlimit and behave reasonably. I think the algorithm in\nbreak_delta_chains() may temporarily store up to uint32_t depth. But I\nthink we won't ever write anything into cur->depth larger than the\nmax-depth limit. So it would probably be sufficient to just check that\nthe --depth argument is reasonably sized and complain otherwise.\n\nI do agree this makes things a bit harder to read, but I think the\nbenefits are pretty measurable. And may make a real usability difference\non a large repository.\n\n-Peff\n"},{"id":"340550","messageId":"CACsJy8Bg_oNj7mJ2PE5AWQ2awf434R9fHXQq-OxA3_0wJN16vw@mail.gmail.com","threadId":"47943","inReplyTo":"20180228101757.GA11803@sigill.intra.peff.net","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-02-28T10:58:50Z","receivedAt":"2018-02-28T10:59:27Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 28, 2018 at 5:17 PM, Jeff King wrote:\n> On Wed, Feb 28, 2018 at 04:27:22PM +0700, Duy Nguyen wrote:\n>\n>> linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n>> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n>> all apps nearly unusuable (granted the problem is partly Linux I/O\n>> scheduler too). So I wonder if we can reduce pack-objects memory\n>> footprint a bit.\n>\n> Yeah, the per object memory footprint is not great. Around 100 million\n> objects it becomes pretty ridiculous. I started to dig into it a year or\n> three ago when I saw such a case, but it turned out to be something that\n> we could prune.\n\nWe could? What could we prune?\n\n> The torvalds/linux fork network has ~23 million objects,\n> so it's probably 7-8 GB of book-keeping. Which is gross, but 64GB in a\n> server isn't uncommon these days.\n\nI wonder if we could just do book keeping for some but not all objects\nbecause all objects simply do not scale. Say we have a big pack of\nmany GBs, could we keep the 80% of its bottom untouched, register the\ntop 20% (mostly non-blobs, and some more blobs as delta base) for\nrepack? We copy the bottom part to the new pack byte-by-byte, then\npack-objects rebuilds the top part with objects from other sources.\n\nThat would of course be less optimal because we can't make delta\nagainst those objects at the bottom. And this makes the assumption\nthat these packs are generated using the same heuristics that we use.\n\n> I think laptops repacking the kernel are probably one of the worst cases\n> (leaving aside the awful Windows repository, but my impression is that\n> they simply can't do a full repack at all there).\n\nYeah. My fear is \"git gc --auto\" kicking in. Even on a laptop we\nshould support working on a repo as big as linux-2.6 (or as smalls as\none from facebook/microsoft perspective). I probably will add a mode\nthat keeps the largest pack alone and just pack the rest in a second\npack in \"git gc --auto\". That might help (I haven't really checked the\ndetails yet, but object_entry book keeping should go down at least)\n\n>> This demonstration patch (probably breaks some tests) would reduce the\n>> size of struct object_entry from from 136 down to 112 bytes on\n>> x86-64. There are 6483999 of these objects, so the saving is 17% or\n>> 148 MB.\n>\n> 136 x 6.5M objects is only about 800MB. I suspect a big chunk of the\n> rest is going to the object structs we create when doing the internal\n> rev-list traversal. And those duplicate the 20-byte sha1s at the very\n> least.\n\nThey are 32 bytes per entry, so it should take less than object_entry.\nI briefly wondered if we should fall back to external rev-list too,\njust to free that memory.\n\nSo about 200 MB for those objects (or maybe more for commits). Add 256\nMB delta cache on top, it's still a bit far from 1.7G. There's\nsomething I'm still missing.\n\n> Another option would be to somehow replace the pack_idx_entry with a\n> reference to a \"struct object\". That would let us at least avoid storing\n> the 20-byte oid twice.\n\n20 bytes saving? /me drools.\n\nPity we can't do the same for 'struct object'. Most of the time we\nhave a giant .idx file with most hashes. We could look up in both\nplaces: the hash table in object.c, and the idx file, to find an\nobject. Then those objects that are associated with .idx file will not\nneed \"oid\" field (needed to as key for the hash table). But I see no\nway to make that change.\n\n>> If we go further, notice that nr_objects is uint32_t, we could convert\n>> the three pointers\n>>\n>> struct object_entry *delta;\n>> struct object_entry *delta_child;\n>> struct object_entry *delta_sibling;\n>>\n>> to\n>>\n>> uint32_t delta;\n>> uint32_t delta_child;\n>> uint32_t delta_sibling;\n>>\n>> which saves 12 bytes (or another 74 MB). 222 MB total is plenty of\n>> space to keep some file cache from being evicted.\n>\n> Yeah, that seems like low-hanging fruit. I'd also note that we don't\n> actually use all of the fields during the whole process. I think some of\n> those delta fields are only used for a short time. So we might be able\n> to reduce peak memory if there are some mutually exclusive bits of each\n> entry (and even if there's some overlap, we'd reduce the length of time\n> we'd need to be at peak).\n\nYeah it looks that way to me too. Now that we have packing_data,\nsplitting object_entry[] to multiple arrays (that is only needed at\nsome point) may be very nice.\n\n>> Is it worth doing this? The struct packing makes it harder to read\n>> (and more fragile too). I added some more artifical limit like max\n>> depth of 2^11. But I think 4096+ depth is getting unreasonable.\n>\n> I'm OK with a limit like 4096, as long as we notice when we hit the\n> limit and behave reasonably. I think the algorithm in\n> break_delta_chains() may temporarily store up to uint32_t depth. But I\n> think we won't ever write anything into cur->depth larger than the\n> max-depth limit. So it would probably be sufficient to just check that\n> the --depth argument is reasonably sized and complain otherwise.\n>\n> I do agree this makes things a bit harder to read, but I think the\n> benefits are pretty measurable. And may make a real usability difference\n> on a large repository.\n\nThanks. I'll go make real patches then (with lots more checks to make\nsure we don't overflow in shortened fields like depth or type).\n-- \nDuy\n"},{"id":"340555","messageId":"20180228111121.GA8925@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8Bg_oNj7mJ2PE5AWQ2awf434R9fHXQq-OxA3_0wJN16vw@mail.gmail.com","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-28T11:11:21Z","receivedAt":"2018-02-28T11:11:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 28, 2018 at 05:58:50PM +0700, Duy Nguyen wrote:\n\n> > Yeah, the per object memory footprint is not great. Around 100 million\n> > objects it becomes pretty ridiculous. I started to dig into it a year or\n> > three ago when I saw such a case, but it turned out to be something that\n> > we could prune.\n> \n> We could? What could we prune?\n\nSorry, I just meant that my 100 million-object case turned out not to\nneed all those objects, and I was able to prune it down. No code fixes\ncame out of it. ;)\n\n> > The torvalds/linux fork network has ~23 million objects,\n> > so it's probably 7-8 GB of book-keeping. Which is gross, but 64GB in a\n> > server isn't uncommon these days.\n> \n> I wonder if we could just do book keeping for some but not all objects\n> because all objects simply do not scale. Say we have a big pack of\n> many GBs, could we keep the 80% of its bottom untouched, register the\n> top 20% (mostly non-blobs, and some more blobs as delta base) for\n> repack? We copy the bottom part to the new pack byte-by-byte, then\n> pack-objects rebuilds the top part with objects from other sources.\n\nYes, though I think it would take a fair bit of surgery to do\ninternally. And some features (like bitmap generation) just wouldn't\nwork at all.\n\nI suspect you could simulate it, though, by just packing your subset\nwith pack-objects (feeding it directly without using \"--revs\") and then\ncatting the resulting packfiles together with a fixed-up header.\n\nAt one point I played with a \"fast pack\" that would just cat packfiles\ntogether. My goal was to make cases with 10,000 packs workable by\ncreating one lousy pack, and then repacking that lousy pack with a\n\"real\" repack. In the end I abandoned it in favor of fixing the\nperformance problems from trying to make a real pack of 10,000 packs. :)\n\nBut I might be able to dig it up if you want to experiment in that\ndirection.\n\n> They are 32 bytes per entry, so it should take less than object_entry.\n> I briefly wondered if we should fall back to external rev-list too,\n> just to free that memory.\n> \n> So about 200 MB for those objects (or maybe more for commits). Add 256\n> MB delta cache on top, it's still a bit far from 1.7G. There's\n> something I'm still missing.\n\nAre you looking at RSS or heap? Keep in mind that you're mmap-ing what's\nprobably a 1GB packfile on disk. If you're under memory pressure that\nwon't all stay resident, but some of it will be counted in RSS.\n\n> Pity we can't do the same for 'struct object'. Most of the time we\n> have a giant .idx file with most hashes. We could look up in both\n> places: the hash table in object.c, and the idx file, to find an\n> object. Then those objects that are associated with .idx file will not\n> need \"oid\" field (needed to as key for the hash table). But I see no\n> way to make that change.\n\nYeah, that would be pretty invasive, I think. I also wonder if it would\nperform worse due to cache effects.\n\n-Peff\n"},{"id":"340556","messageId":"CACsJy8ASFF-82jcm3nNBsjFGpxwUNJZT+3kE+buuZt0mtG6KfA@mail.gmail.com","threadId":"47943","inReplyTo":"20180228111121.GA8925@sigill.intra.peff.net","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-02-28T11:24:27Z","receivedAt":"2018-02-28T11:25:03Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 28, 2018 at 6:11 PM, Jeff King wrote:\n>> > The torvalds/linux fork network has ~23 million objects,\n>> > so it's probably 7-8 GB of book-keeping. Which is gross, but 64GB in a\n>> > server isn't uncommon these days.\n>>\n>> I wonder if we could just do book keeping for some but not all objects\n>> because all objects simply do not scale. Say we have a big pack of\n>> many GBs, could we keep the 80% of its bottom untouched, register the\n>> top 20% (mostly non-blobs, and some more blobs as delta base) for\n>> repack? We copy the bottom part to the new pack byte-by-byte, then\n>> pack-objects rebuilds the top part with objects from other sources.\n>\n> Yes, though I think it would take a fair bit of surgery to do\n> internally. And some features (like bitmap generation) just wouldn't\n> work at all.\n>\n> I suspect you could simulate it, though, by just packing your subset\n> with pack-objects (feeding it directly without using \"--revs\") and then\n> catting the resulting packfiles together with a fixed-up header.\n>\n> At one point I played with a \"fast pack\" that would just cat packfiles\n> together. My goal was to make cases with 10,000 packs workable by\n> creating one lousy pack, and then repacking that lousy pack with a\n> \"real\" repack. In the end I abandoned it in favor of fixing the\n> performance problems from trying to make a real pack of 10,000 packs. :)\n>\n> But I might be able to dig it up if you want to experiment in that\n> direction.\n\nNaah it's ok. I'll go similar direction, but I'd repack those pack\nfiles too except the big one. Let's see how it turns out.\n\n>> They are 32 bytes per entry, so it should take less than object_entry.\n>> I briefly wondered if we should fall back to external rev-list too,\n>> just to free that memory.\n>>\n>> So about 200 MB for those objects (or maybe more for commits). Add 256\n>> MB delta cache on top, it's still a bit far from 1.7G. There's\n>> something I'm still missing.\n>\n> Are you looking at RSS or heap? Keep in mind that you're mmap-ing what's\n> probably a 1GB packfile on disk. If you're under memory pressure that\n> won't all stay resident, but some of it will be counted in RSS.\n\nInteresting. It was RSS.\n\n>> Pity we can't do the same for 'struct object'. Most of the time we\n>> have a giant .idx file with most hashes. We could look up in both\n>> places: the hash table in object.c, and the idx file, to find an\n>> object. Then those objects that are associated with .idx file will not\n>> need \"oid\" field (needed to as key for the hash table). But I see no\n>> way to make that change.\n>\n> Yeah, that would be pretty invasive, I think. I also wonder if it would\n> perform worse due to cache effects.\n\nIt should be better because of cache effects, I think. I mean, hash\nmap is the least cache friendly lookup. Moving most objects out of the\nhash table shrinks it, which is even nicer to cache. But we also lose\nO(1) when we do binary search on .idx file (after failing to find the\nsame object in the hash table)\n-- \nDuy\n"},{"id":"340590","messageId":"20180228182233.GC15256@dcvr","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2018-02-28T18:22:33Z","receivedAt":"2018-02-28T18:22:38Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Duy Nguyen wrote:\n> which saves 12 bytes (or another 74 MB). 222 MB total is plenty of\n> space to keep some file cache from being evicted.\n\nNice! I can definitely benefit from lower memory usage when\npacking. Fwiw, I use pahole with other projects to help find\npacking opportunities:\n\n\tgit://git.kernel.org/pub/scm/devel/pahole/pahole.git\n\n> @@ -14,11 +26,10 @@ struct object_entry {\n> \tvoid *delta_data;\t/* cached delta (uncompressed) */\n> \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n> \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n> -\tenum object_type type;\n> -\tenum object_type in_pack_type;\t/* could be delta */\n> \tuint32_t hash;\t\t\t/* name hint hash */\n> -\tunsigned int in_pack_pos;\n> \tunsigned char in_pack_header_size;\n> +\tunsigned type:3;\t /* enum object_type */\n> +\tunsigned in_pack_type:3; /* enum object_type - could be delta */\n\nFor C99 compilers, enums can be bitfields. I introduced the\nfollowing macro into Ruby a few weeks ago to remain compatible\nwith non-C99 compilers:\n\n/*\n * For declaring bitfields out of non-unsigned int types:\n * struct date {\n * BITFIELD(enum months) month:4;\n * ...\n * };\n */\n#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)\n# define BITFIELD(type) type\n#else\n# define BITFIELD(type) unsigned int\n#endif\n"},{"id":"340694","messageId":"20180301090039.GA19167@ash","threadId":"47943","inReplyTo":"20180228182233.GC15256@dcvr","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:00:39Z","receivedAt":"2018-03-01T09:00:52Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 28, 2018 at 06:22:33PM +0000, Eric Wong wrote:\n> Duy Nguyen wrote:\n> > which saves 12 bytes (or another 74 MB). 222 MB total is plenty of\n> > space to keep some file cache from being evicted.\n> \n> Nice! I can definitely benefit from lower memory usage when\n> packing. Fwiw, I use pahole with other projects to help find\n> packing opportunities:\n> \n> \tgit://git.kernel.org/pub/scm/devel/pahole/pahole.git\n\nYes it's a wonderful tool.\n\n> > @@ -14,11 +26,10 @@ struct object_entry {\n> > \tvoid *delta_data;\t/* cached delta (uncompressed) */\n> > \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n> > \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n> > -\tenum object_type type;\n> > -\tenum object_type in_pack_type;\t/* could be delta */\n> > \tuint32_t hash;\t\t\t/* name hint hash */\n> > -\tunsigned int in_pack_pos;\n> > \tunsigned char in_pack_header_size;\n> > +\tunsigned type:3;\t /* enum object_type */\n> > +\tunsigned in_pack_type:3; /* enum object_type - could be delta */\n> \n> For C99 compilers, enums can be bitfields. I introduced the\n> following macro into Ruby a few weeks ago to remain compatible\n> with non-C99 compilers:\n> \n> /*\n> * For declaring bitfields out of non-unsigned int types:\n> * struct date {\n> * BITFIELD(enum months) month:4;\n> * ...\n> * };\n> */\n> #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)\n> # define BITFIELD(type) type\n> #else\n> # define BITFIELD(type) unsigned int\n> #endif\n\nI tried this and got\n\nIn file included from builtin/pack-objects.c:20:0:\n./pack-objects.h:49:19: l?i: ?type? is narrower than values of its type [-Werror]\n enum object_type type:TYPE_BITS;\n ^~~~\n\nThe compiler is not wrong. What it does not realize is pack-objects\ncode never uses out-of-range values (OBJ_BAD and OBJ_ANY) but I don't\nsee how I could suppress this warning. So I went back to non-enum\nbitfields.\n\n--\nDuy\n"},{"id":"340695","messageId":"20180301091052.32267-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 01/11] pack-objects: document holes in struct object_entry.h","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:42Z","receivedAt":"2018-03-01T09:11:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..720a8e8756 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -28,6 +28,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n+\t/* XXX 28 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -40,6 +41,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n+\t/* size: 136, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340696","messageId":"20180301091052.32267-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 02/11] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:43Z","receivedAt":"2018-03-01T09:11:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This saves 8 bytes in sizeof(struct object_entry). On a large\nrepository like linux-2.6.git (6.5M objects), this saves us 52MB\nmemory.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 14 ++++++++++++--\n cache.h | 2 ++\n object.h | 1 -\n pack-objects.h | 8 ++++----\n 4 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..fd217cb51f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n@@ -1580,6 +1586,10 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t */\n \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n \t\t\t\t\t &entry->size);\n+\t} else {\n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->type = type;\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 720a8e8756..f8b06e2521 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -14,11 +14,11 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n+\tunsigned type:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n@@ -28,7 +28,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n-\t/* XXX 28 bits hole, try to pack */\n+\t/* XXX 22 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -41,7 +41,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n-\t/* size: 136, padding: 4 */\n+\t/* size: 128, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340697","messageId":"20180301091052.32267-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 03/11] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:44Z","receivedAt":"2018-03-01T09:11:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 33 ++++++++++++++++++++-------------\n 2 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fd217cb51f..a4dbb40824 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex f8b06e2521..fca334ab4d 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS 2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n@@ -27,21 +42,13 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 20 bits hole, try to pack */\n \n-\t/* XXX 22 bits hole, try to pack */\n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n-\t/* size: 128, padding: 4 */\n+\n+\t/* size: 120, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340698","messageId":"20180301091052.32267-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:45Z","receivedAt":"2018-03-01T09:11:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This does not give us any saving due to padding. But we will be able\nto save once we cut 4 bytes out of this struct in a subsequent patch.\n\nBecause of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 6 ++----\n 2 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a4dbb40824..cfd97da7db 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth > (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS));\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex fca334ab4d..3941e6c9a6 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS 2\n+#define OE_DEPTH_BITS 12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -43,10 +44,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\t/* XXX 20 bits hole, try to pack */\n-\n-\tint depth;\n+\tunsigned depth:OE_DEPTH_BITS;\n \n \t/* size: 120, padding: 4 */\n };\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340699","messageId":"20180301091052.32267-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 06/11] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:47Z","receivedAt":"2018-03-01T09:11:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot). This saves us 8 bytes in struct object_entry.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 8 ++++++--\n 5 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex cfd97da7db..7bb5544883 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -878,7 +878,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..1360a93311 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\tIN_PACK_POS(to_pack, entry) = i;\n \n \t\tswitch (entry->type) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn IN_PACK_POS(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..f21479fe16 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = IN_PACK_POS(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 017cc3425f..3bef28196c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,9 @@\n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n \n+#define IN_PACK_POS(to_pack, obj) \\\n+\t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n+\n /*\n * State flags for depth-first search used for analyzing delta cycles.\n *\n@@ -31,7 +34,6 @@ struct object_entry {\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -46,7 +48,7 @@ struct object_entry {\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 120, padding: 4 */\n+\t/* size: 112 */\n };\n \n struct packing_data {\n@@ -55,6 +57,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340700","messageId":"20180301091052.32267-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 08/11] pack-objects: faster reverse packed_git lookup","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:49Z","receivedAt":"2018-03-01T09:11:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We do a linear search for in_pack index in create_object_entry(). This\nfunction is called for every available object in the worst case (and\non linux-2.6.git, that's about 6.5M). Try to avoid that by saving the\nindex in packed_git. Since we should not have zillions of packs, this\nextra space should not be a big deal.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 11 ++---------\n cache.h | 1 +\n 2 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d0d371714a..1fdb85ebb5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1074,15 +1074,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tint i;\n-\n-\t\tfor (i = 0; i < (1 << OE_IN_PACK_BITS); i++)\n-\t\t\tif (to_pack.in_pack[i] == found_pack) {\n-\t\t\t\tentry->in_pack_idx = i;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\tif (i == (1 << OE_IN_PACK_BITS))\n-\t\t\tdie(\"BUG: pack not found!\");\n+\t\tentry->in_pack_idx = found_pack->index;\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -2980,6 +2972,7 @@ static void init_in_pack_mapping(struct packing_data *to_pack)\n \t\tif (i >= (1 << OE_IN_PACK_BITS))\n \t\t\tdie(\"BUG: too many packs to handle!\");\n \t\tto_pack->in_pack[i] = p;\n+\t\tp->index = i;\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340701","messageId":"20180301091052.32267-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 05/11] pack-objects: note about in_pack_header_size","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:46Z","receivedAt":"2018-03-01T09:11:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Object header in a pack is packed really tight (see\npack-format.txt). Even with 8 bytes length, we need 9-10 bytes most,\nplus a hash (20 bytes). Which means this field only needs to store a\nnumber as big as 32 (5 bits).\n\nThis is trickier to pack tight though since a new hash algorithm is\ncoming, the number of bits needed may quickly increase. So leave it\nfor now.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 3941e6c9a6..017cc3425f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -32,7 +32,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n-\tunsigned char in_pack_header_size;\n+\tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340702","messageId":"20180301091052.32267-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 07/11] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:48Z","receivedAt":"2018-03-01T09:12:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index isntead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 256 (still\nunreasonably high for a repo to work well). If you have more than 256\npacks, you'll need an older version of Git to repack first.\n\nThis technically saves 7 bytes. But we don't see any of that in\npractice due to padding. The saving becomes real when we pack this\nstruct tighter later.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 48 ++++++++++++++++++++++++++++++++----------\n pack-objects.h | 12 +++++++++--\n 2 files changed, 47 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7bb5544883..d0d371714a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -367,7 +367,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +478,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(&to_pack, entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n@@ -1074,7 +1074,15 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < (1 << OE_IN_PACK_BITS); i++)\n+\t\t\tif (to_pack.in_pack[i] == found_pack) {\n+\t\t\t\tentry->in_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\tif (i == (1 << OE_IN_PACK_BITS))\n+\t\t\tdie(\"BUG: pack not found!\");\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1407,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(&to_pack, entry)) {\n+\t\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1543,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(&to_pack, a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(&to_pack, b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1588,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(&to_pack, entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1858,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(&to_pack, trg_entry) &&\n+\t IN_PACK(&to_pack, trg_entry) == IN_PACK(&to_pack, src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -2958,6 +2968,21 @@ static int option_parse_unpack_unreachable(const struct option *opt,\n \treturn 0;\n }\n \n+static void init_in_pack_mapping(struct packing_data *to_pack)\n+{\n+\tstruct packed_git *p;\n+\tint i = 0;\n+\n+\t/* let IN_PACK() return NULL if in_pack_idx is zero */\n+\tto_pack->in_pack[i++] = NULL;\n+\n+\tfor (p = packed_git; p; p = p->next, i++) {\n+\t\tif (i >= (1 << OE_IN_PACK_BITS))\n+\t\t\tdie(\"BUG: too many packs to handle!\");\n+\t\tto_pack->in_pack[i] = p;\n+\t}\n+}\n+\n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n@@ -3190,6 +3215,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t}\n \t}\n+\tinit_in_pack_mapping(&to_pack);\n \n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 3bef28196c..839d5dc4fd 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,10 +3,14 @@\n \n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n+#define OE_IN_PACK_BITS 8\n \n #define IN_PACK_POS(to_pack, obj) \\\n \t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n \n+#define IN_PACK(to_pack, obj) \\\n+\t(to_pack)->in_pack[(obj)->in_pack_idx]\n+\n /*\n * State flags for depth-first search used for analyzing delta cycles.\n *\n@@ -23,7 +27,6 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -35,6 +38,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n@@ -46,9 +50,12 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 12 bits hole, try to pack */\n+\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112 */\n+\t/* size: 112, padding: 4 */\n };\n \n struct packing_data {\n@@ -59,6 +66,7 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340703","messageId":"20180301091052.32267-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 09/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:50Z","receivedAt":"2018-03-01T09:12:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Notice that packing_data::nr_objects is uint32_t, we could only handle\nmaximum 4G objects and can address all of them with an uint32_t. If we\nuse a pointer here, we waste 4 bytes on 64 bit architecture.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\nThere are holes in this struct but this patch is already big. Struct\npacking can be done separately. Even with holes, we save 8 bytes per\nobject_entry.\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 124 +++++++++++++++++++++++------------------\n pack-objects.h | 14 +++--\n 2 files changed, 78 insertions(+), 60 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1fdb85ebb5..45076f2523 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,20 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define DELTA(obj) \\\n+\t((obj)->delta_idx ? &to_pack.objects[(obj)->delta_idx - 1] : NULL)\n+#define DELTA_CHILD(obj) \\\n+\t((obj)->delta_child_idx ? &to_pack.objects[(obj)->delta_child_idx - 1] : NULL)\n+#define DELTA_SIBLING(obj) \\\n+\t((obj)->delta_sibling_idx ? &to_pack.objects[(obj)->delta_sibling_idx - 1] : NULL)\n+\n+#define CLEAR_DELTA(obj) (obj)->delta_idx = 0\n+#define CLEAR_DELTA_CHILD(obj) (obj)->delta_child_idx = 0\n+#define CLEAR_DELTA_SIBLING(obj) (obj)->delta_sibling_idx = 0\n+\n+#define SET_DELTA(obj, val) (obj)->delta_idx = ((val) - to_pack.objects) + 1\n+#define SET_DELTA_CHILD(obj, val) (obj)->delta_child_idx = ((val) - to_pack.objects) + 1\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -125,11 +139,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -286,12 +300,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -315,7 +329,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -341,7 +355,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -377,8 +391,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -406,7 +420,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -425,7 +439,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -465,13 +479,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -486,7 +500,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (entry->type != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -538,12 +552,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tCLEAR_DELTA(e);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -605,34 +619,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -643,7 +657,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -658,8 +672,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tCLEAR_DELTA_CHILD(&objects[i]);\n+\t\tCLEAR_DELTA_SIBLING(&objects[i]);\n \t}\n \n \t/*\n@@ -669,11 +683,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1491,10 +1505,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\tentry->type = entry->in_pack_type;\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1565,17 +1579,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tCLEAR_DELTA(entry);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1617,7 +1633,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1642,7 +1658,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1665,7 +1681,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1680,7 +1696,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1863,7 +1879,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1939,7 +1955,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1968,7 +1984,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1977,13 +1993,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2052,7 +2068,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2102,7 +2118,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2110,7 +2126,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2431,7 +2447,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 839d5dc4fd..f339f0411a 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -28,11 +28,13 @@ struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n+\t/* XXX 4 bytes hole, try to pack */\n+\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -55,7 +57,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112, padding: 4 */\n+\t/* size: 104, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340704","messageId":"20180301091052.32267-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 10/11] pack-objects: reorder 'hash' to pack struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:51Z","receivedAt":"2018-03-01T09:12:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex f339f0411a..52087b32e5 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -33,12 +33,10 @@ struct object_entry {\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\t/* XXX 4 bytes hole, try to pack */\n-\n+\tuint32_t hash;\t\t\t/* name hint hash */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -57,7 +55,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 104, padding: 4 */\n+\t/* size: 96 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340705","messageId":"20180301091052.32267-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH 11/11] pack-objects: increase pack file limit to 4096","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:52Z","receivedAt":"2018-03-01T09:12:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When OE_IN_PACK_BITS was added, we didn't have many bits left to spare\nso the max number of packs that pack-objects could handle was limited\nto 256. Now we have more spare bits, let's increase it to 4096 to be\non the safe side. If you have more than this many packs, you may need\nto reconsider if you're still sane.\n\nIncreasing this also increases memory a bit because in_pack[] array in\npacking_data is bigger, roughly 32kb, which is insignificant in\npack-objects context.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 52087b32e5..ec4eba4ee4 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,7 +3,7 @@\n \n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n-#define OE_IN_PACK_BITS 8\n+#define OE_IN_PACK_BITS 12\n \n #define IN_PACK_POS(to_pack, obj) \\\n \t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n@@ -24,6 +24,11 @@ enum dfs_state {\n \tDFS_NUM_STATES\n };\n \n+/*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n@@ -51,7 +56,7 @@ struct object_entry {\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/* XXX 12 bits hole, try to pack */\n+\t/* XXX 8 bits hole, try to pack */\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340706","messageId":"20180301091052.32267-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"[PATCH 00/11] Reduce pack-objects memory footprint","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:10:41Z","receivedAt":"2018-03-01T09:13:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The array of object_entry in pack-objects can take a lot of memory\nwhen pack-objects is run in \"pack everything\" mode. On linux-2.6.git,\nthis array alone takes roughly 800MB.\n\nThis series reorders some fields and reduces field size... to keep\nthis struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on\n64-bit linux and saves 260MB on linux-2.6.git.\n\nNow the bad side:\n\n- the number of pack files pack-objects can handle is reduced to 4096\n (previously unlimited)\n- max delta chain is also limited to 4096 (previously practically\n unlimited)\n- some patches are quite invasive (e.g. replacing pointer with\n uint32_t) and reduces readability a bit.\n- it may be tricker to add more data in object_entry in the future.\n\nNguyễn Thái Ngọc Duy (11):\n pack-objects: document holes in struct object_entry.h\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: note about in_pack_header_size\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: faster reverse packed_git lookup\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: reorder 'hash' to pack struct object_entry\n pack-objects: increase pack file limit to 4096\n\n builtin/pack-objects.c | 189 ++++++++++++++++++++++++++---------------\n cache.h | 3 +\n object.h | 1 -\n pack-bitmap-write.c | 8 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.h | 70 ++++++++++-----\n 7 files changed, 180 insertions(+), 97 deletions(-)\n\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340707","messageId":"20180301092046.2769-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"[PATCH/RFC 0/1] Avoid expensive 'repack -ad' in gc --auto","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:20:45Z","receivedAt":"2018-03-01T09:21:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The series [1] I just sent helps reduce pack-objects memory footprint\na bit. But even then it's still a huge memory hog. So this patch makes\na special treatment for gc --auto: avoid it completely.\n\nThe trick here is not new (pinning the largest pack with a .keep\nfile). It's just never done automatically. I think this is a good\nthing to do, provided that gc --auto estimates memory usage more or\nless correct.\n\nAnd \"git gc --auto\" should run even on weak machines because it's part\nof regular repo maintenance. You can't tell people \"You can't work on\nlinux-2.6.git repository because your machine has too little memory\".\n\nThe only thing left I think I should do is to use an external rev-list\nto free up some more memory. But let's see how the first patch goes\nfirst (documents and tests are missing, I know).\n\n[1] https://public-inbox.org/git/%3C20180301091052.32267-1-pclouds@gmail.com%3E/\n\nNguyễn Thái Ngọc Duy (1):\n gc --auto: exclude the largest giant pack in low-memory config\n\n builtin/gc.c | 125 +++++++++++++++++++++++++++++++++++++++--\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n 5 files changed, 128 insertions(+), 6 deletions(-)\n\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340708","messageId":"20180301092046.2769-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301092046.2769-1-pclouds@gmail.com","subject":"[PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-01T09:20:46Z","receivedAt":"2018-03-01T09:21:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\nlargest pack to avoid this problem is also known for a long time.\n\nLet's do the suggestion automatically instead of waiting for people to\ncome to Git mailing list and get the advice. When a certain condition\nis met, gc --auto create a .keep file temporary before repack is run,\nthen remove it afterward.\n\ngc --auto does this based on an estimation of pack-objects memory\nusage and whether that fits in one third of system memory (the\nassumption here is for desktop environment where there are many other\napplications running).\n\nSince the estimation may be inaccurate and that 1/3 threshold is\narbitrary, give the user a finer control over this mechanism as well:\nif the largest pack is larger than gc.bigPackThreshold, it's kept.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/gc.c | 125 +++++++++++++++++++++++++++++++++++++++--\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n 5 files changed, 128 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..2d9965bcdf 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,10 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -39,6 +43,8 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_pack_threshold = 0;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -49,6 +55,7 @@ static struct argv_array rerere = ARGV_ARRAY_INIT;\n \n static struct tempfile *pidfile;\n static struct lock_file log_lock;\n+static struct strbuf temp_keep_file = STRBUF_INIT;\n \n static struct string_list pack_garbage = STRING_LIST_INIT_DUP;\n \n@@ -93,6 +100,18 @@ static void process_log_file(void)\n \t}\n }\n \n+static void delete_temp_keep_file(void)\n+{\n+\tunlink(temp_keep_file.buf);\n+}\n+\n+static void delete_temp_keep_file_on_signal(int signo)\n+{\n+\tdelete_temp_keep_file();\n+\tsigchain_pop(signo);\n+\traise(signo);\n+}\n+\n static void process_log_file_at_exit(void)\n {\n \tfflush(stderr);\n@@ -126,6 +145,9 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -164,7 +186,7 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static int too_many_packs(void)\n+static int too_many_packs(struct packed_git **largest_pack)\n {\n \tstruct packed_git *p;\n \tint cnt;\n@@ -173,22 +195,104 @@ static int too_many_packs(void)\n \t\treturn 0;\n \n \tprepare_packed_git();\n+\t*largest_pack = NULL;\n \tfor (cnt = 0, p = packed_git; p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n \t\t\tcontinue;\n+\t\tif (!*largest_pack || (*largest_pack)->pack_size < p->pack_size)\n+\t\t\t*largest_pack = p;\n \t\t/*\n \t\t * Perhaps check the size of the pack and count only\n \t\t * very small ones here?\n \t\t */\n \t\tcnt++;\n \t}\n+\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static inline unsigned long total_ram(void)\n+{\n+\tunsigned long default_ram = 4;\n+#ifdef HAVE_SYSINFO\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tint length;\n+\n+\tmib[0] = CTL_HW;\n+\tmib[1] = HW_MEMSIZE;\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo;ullTotalPhys;\n+#else\n+\tfprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n+\t\tdefault_ram);\n+#endif\n+\treturn default_ram * 1024 * 1024 * 1024;\n+}\n+\n+static int pack_objects_uses_too_much_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t mem_want, mem_have;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\tif (big_pack_threshold)\n+\t\treturn pack->pack_size >= big_pack_threshold;\n+\n+\t/* First we have to scan through at least one pack */\n+\tmem_want = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\tmem_want += sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\tmem_want += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\tmem_want += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\tmem_want += sizeof(struct object *) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\tmem_want += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\tmem_want += max_delta_cache_size;\n+\n+\t/* Only allow 1/3 of memory for pack-objects */\n+\tmem_have = total_ram() / 3;\n+\n+\treturn mem_want >= mem_have;\n+\n+}\n+\n+static void add_repack_all_option(struct packed_git *exclude_pack)\n {\n+\tif (pack_objects_uses_too_much_memory(exclude_pack)) {\n+\t\tstrbuf_addstr(&temp_keep_file, exclude_pack->pack_name);\n+\t\tstrbuf_strip_suffix(&temp_keep_file, \".pack\");\n+\t\tstrbuf_addstr(&temp_keep_file, \".keep\");\n+\t}\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n \telse {\n@@ -205,6 +309,7 @@ static void add_repack_incremental_option(void)\n \n static int need_to_gc(void)\n {\n+\tstruct packed_git *largest_pack = NULL;\n \t/*\n \t * Setting gc.auto to 0 or negative can disable the\n \t * automatic gc.\n@@ -218,8 +323,8 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option();\n+\tif (too_many_packs(&largest_pack))\n+\t\tadd_repack_all_option(largest_pack);\n \telse if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -428,7 +533,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tdaemonized = !daemonize();\n \t\t}\n \t} else\n-\t\tadd_repack_all_option();\n+\t\tadd_repack_all_option(NULL);\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n@@ -450,6 +555,16 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (gc_before_repack())\n \t\treturn -1;\n \n+\tif (temp_keep_file.len) {\n+\t\tint fd = open(temp_keep_file.buf, O_CREAT | O_RDWR, 0644);\n+\t\tif (fd != -1) {\n+\t\t\tsigchain_push_common(delete_temp_keep_file_on_signal);\n+\t\t\tatexit(delete_temp_keep_file);\n+\t\t\tclose(fd);\n+\t\t} else {\n+\t\t\tstrbuf_release(&temp_keep_file);\n+\t\t}\n+\t}\n \tif (!repository_format_precious_objects) {\n \t\tif (run_command_v_opt(repack.argv, RUN_GIT_CMD))\n \t\t\treturn error(FAILED_RUN, repack.argv[0]);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..722cc999dc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -78,7 +78,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 68b2ad531e..a84b21986d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340723","messageId":"87zi3sc6q1.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-8-pclouds@gmail.com","subject":"Re: [PATCH 07/11] pack-objects: move in_pack out of struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-01T12:37:10Z","receivedAt":"2018-03-01T12:37:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> pack. Use an index isntead since the number of packs should be\n\ns/isntead/instead/\n\n> This limits the number of packs we can handle to 256 (still\n> unreasonably high for a repo to work well). If you have more than 256\n> packs, you'll need an older version of Git to repack first.\n\nSo if you have gc.autoPackLimit=300 this will break, how does it break?\n\nShould we also make (& document) setting that variable higher than 256\nan error?\n"},{"id":"340724","messageId":"87y3jbdiof.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"Re: [PATCH 00/11] Reduce pack-objects memory footprint","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-01T13:33:36Z","receivedAt":"2018-03-01T13:33:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> The array of object_entry in pack-objects can take a lot of memory\n> when pack-objects is run in \"pack everything\" mode. On linux-2.6.git,\n> this array alone takes roughly 800MB.\n>\n> This series reorders some fields and reduces field size... to keep\n> this struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on\n> 64-bit linux and saves 260MB on linux-2.6.git.\n\nI'm very interested in this patch series. I don't have time to test this\none right now (have to run), but with your previous RFC patch memory use\n(in the ~4GB range) on a big in-house repo went down by a bit over 3%,\nand it's ~5% faster.\n\nBefore/after RSS 4440812 / 4290000 & runtime 172.73 / 162.45. This is\nafter having already done a full git gc before, data via /usr/bin/time\n-v.\n\nSo not huge, but respectable.\n\nWe have a big repo, and this gets repacked on 6-8GB of memory on dev\nKVMs, so we're under a fair bit of memory pressure. git-gc slows things\ndown a lot.\n\nIt would be really nice to have something that made it use drastically\nless memory at the cost of less efficient packs. Is the property that\nyou need to spend give or take the size of .git/objects in memory\nsomething inherent, or just a limitation of the current implementation?\nI.e. could we do a first pass to pick some objects based on some\nheuristic, then repack them N at a time, and finally delete the\nnow-obsolete packs?\n\nAnother thing I've dealt with is that on these machines their\nNFS-mounted storage gets exhausted (I'm told) due to some pathological\noperations git does during repack, I/O tends to get 5-6x slower. Of\ncourse ionice doesn't help because the local kernel doesn't know\nanything about how harmful it is.\n"},{"id":"340728","messageId":"20180301144942.GA837@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180301091052.32267-8-pclouds@gmail.com","subject":"Re: [PATCH 07/11] pack-objects: move in_pack out of struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-01T14:49:42Z","receivedAt":"2018-03-01T14:49:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 01, 2018 at 04:10:48PM +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\n> pack. Use an index isntead since the number of packs should be\n> relatively small.\n> \n> This limits the number of packs we can handle to 256 (still\n> unreasonably high for a repo to work well). If you have more than 256\n> packs, you'll need an older version of Git to repack first.\n\nI overall like the direction of this series, but I think this one is\njust too much. While you definitely shouldn't have a ton of packs, this\nleaves the user with no real escape hatch. And 256 isn't actually that\nmany packs.\n\n-Peff\n"},{"id":"340738","messageId":"xmqq4llziskr.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180301091052.32267-5-pclouds@gmail.com","subject":"Re: [PATCH 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T18:00:52Z","receivedAt":"2018-03-01T18:01:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This does not give us any saving due to padding. But we will be able\n> to save once we cut 4 bytes out of this struct in a subsequent patch.\n>\n> Because of struct packing from now on we can only handle max depth\n> 4095 (or even lower when new booleans are added in this struct). This\n> should be ok since long delta chain will cause significant slow down\n> anyway.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\nThis should be marked as RFC/PATCH; I do not have objection to\nlimiting the delta depth to some reasonable length (rather than the\ncurrent 1<<31 or worse 1<<63), and 4k may be such a reasonable limit\n(I'd actually think anything more than a few hundreds is probably a\nbad idea), but it needs to be documented.\n\n> builtin/pack-objects.c | 4 ++++\n> pack-objects.h | 6 ++----\n> 2 files changed, 6 insertions(+), 4 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index a4dbb40824..cfd97da7db 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (pack_to_stdout != !base_name || argc)\n> \t\tusage_with_options(pack_usage, pack_objects_options);\n> \n> +\tif (depth > (1 << OE_DEPTH_BITS))\n> +\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n> +\t\t depth, (1 << OE_DEPTH_BITS));\n> +\n> \targv_array_push(&rp, \"pack-objects\");\n> \tif (thin) {\n> \t\tuse_internal_rev_list = 1;\n> diff --git a/pack-objects.h b/pack-objects.h\n> index fca334ab4d..3941e6c9a6 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -2,6 +2,7 @@\n> #define PACK_OBJECTS_H\n> \n> #define OE_DFS_STATE_BITS 2\n> +#define OE_DEPTH_BITS 12\n> \n> /*\n> * State flags for depth-first search used for analyzing delta cycles.\n> @@ -43,10 +44,7 @@ struct object_entry {\n> \tunsigned tagged:1; /* near the very tip of refs */\n> \tunsigned filled:1; /* assigned write-order */\n> \tunsigned dfs_state:OE_DFS_STATE_BITS;\n> -\n> -\t/* XXX 20 bits hole, try to pack */\n> -\n> -\tint depth;\n> +\tunsigned depth:OE_DEPTH_BITS;\n> \n> \t/* size: 120, padding: 4 */\n> };\n"},{"id":"340739","messageId":"xmqqzi3rhdsu.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180301091052.32267-8-pclouds@gmail.com","subject":"Re: [PATCH 07/11] pack-objects: move in_pack out of struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T18:05:21Z","receivedAt":"2018-03-01T18:05:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\n> pack. Use an index isntead since the number of packs should be\n> relatively small.\n>\n> This limits the number of packs we can handle to 256 (still\n> unreasonably high for a repo to work well). If you have more than 256\n> packs, you'll need an older version of Git to repack first.\n\nI can tell without looking at the rest of the thread that people had\nreasonable objection against this stance ;-) This will not fly well.\n"},{"id":"340740","messageId":"xmqqvaefhdoc.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180301091052.32267-10-pclouds@gmail.com","subject":"Re: [PATCH 09/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T18:08:03Z","receivedAt":"2018-03-01T18:08:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Notice that packing_data::nr_objects is uint32_t, we could only handle\n> maximum 4G objects and can address all of them with an uint32_t. If we\n> use a pointer here, we waste 4 bytes on 64 bit architecture.\n>\n> Convert these delta pointers to indexes. Since we need to handle NULL\n> pointers as well, the index is shifted by one [1].\n\nMakes perfect sense.\n\nI do not think losing 1 slot out of possible 4G is a regression,\nunlike the 256 packfile limit 07/11 imposes.\n"},{"id":"340742","messageId":"xmqqpo4nhddm.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180301092046.2769-2-pclouds@gmail.com","subject":"Re: [PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T18:14:29Z","receivedAt":"2018-03-01T18:14:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> pack-objects could be a big memory hog especially on large repos,\n> everybody knows that. The suggestion to stick a .keep file on the\n> largest pack to avoid this problem is also known for a long time.\n\nYup, but not that it is not \"largest\" per-se. The thing being large\nis a mere consequence that it is the base pack that holds the bulk\nof older parts of the history (e.g. the one that you obtained via\nthe initial clone).\n\n> Let's do the suggestion automatically instead of waiting for people to\n> come to Git mailing list and get the advice. When a certain condition\n> is met, gc --auto create a .keep file temporary before repack is run,\n> then remove it afterward.\n>\n> gc --auto does this based on an estimation of pack-objects memory\n> usage and whether that fits in one third of system memory (the\n> assumption here is for desktop environment where there are many other\n> applications running).\n>\n> Since the estimation may be inaccurate and that 1/3 threshold is\n> arbitrary, give the user a finer control over this mechanism as well:\n> if the largest pack is larger than gc.bigPackThreshold, it's kept.\n\nIf this is a transient mechanism during a single gc session, it\nwould be far more preferrable if we can find a way to do this\nwithout actually having a .keep file on the filesystem.\n\n"},{"id":"340771","messageId":"CACsJy8BSW_DaHRYDVeL=iHsR=4+My1GYRofBVsFALn0=D4zsGw@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqpo4nhddm.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T00:00:16Z","receivedAt":"2018-03-02T00:00:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 2, 2018 at 1:14 AM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> pack-objects could be a big memory hog especially on large repos,\n>> everybody knows that. The suggestion to stick a .keep file on the\n>> largest pack to avoid this problem is also known for a long time.\n>\n> Yup, but not that it is not \"largest\" per-se. The thing being large\n> is a mere consequence that it is the base pack that holds the bulk\n> of older parts of the history (e.g. the one that you obtained via\n> the initial clone).\n\nThanks, \"base pack\" sounds much better. I was having trouble with\nwording because I didn't nail this one down.\n\n>> Let's do the suggestion automatically instead of waiting for people to\n>> come to Git mailing list and get the advice. When a certain condition\n>> is met, gc --auto create a .keep file temporary before repack is run,\n>> then remove it afterward.\n>>\n>> gc --auto does this based on an estimation of pack-objects memory\n>> usage and whether that fits in one third of system memory (the\n>> assumption here is for desktop environment where there are many other\n>> applications running).\n>>\n>> Since the estimation may be inaccurate and that 1/3 threshold is\n>> arbitrary, give the user a finer control over this mechanism as well:\n>> if the largest pack is larger than gc.bigPackThreshold, it's kept.\n>\n> If this is a transient mechanism during a single gc session, it\n> would be far more preferrable if we can find a way to do this\n> without actually having a .keep file on the filesystem.\n\nThat was my first attempt, manipulating packed_git::pack_keep inside\npack-objects. Then my whole git.git was gone. I was scared off so I\ndid this instead.\n\nI've learned my lesson though (never test dangerous operations on your\nworktree!) and will do that pack_keep again _if_ this gc --auto still\nsounds like a good direction to go.\n-- \nDuy\n"},{"id":"340772","messageId":"CACsJy8D1bRB2qkxmUhaZU3DLA-NF+D5Zqp+1tyUZqmEw88T0ig@mail.gmail.com","threadId":"47943","inReplyTo":"20180301144942.GA837@sigill.intra.peff.net","subject":"Re: [PATCH 07/11] pack-objects: move in_pack out of struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T00:02:31Z","receivedAt":"2018-03-02T00:03:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 1, 2018 at 9:49 PM, Jeff King wrote:\n> On Thu, Mar 01, 2018 at 04:10:48PM +0700, Nguyễn Thái Ngọc Duy wrote:\n>\n>> Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\n>> pack. Use an index isntead since the number of packs should be\n>> relatively small.\n>>\n>> This limits the number of packs we can handle to 256 (still\n>> unreasonably high for a repo to work well). If you have more than 256\n>> packs, you'll need an older version of Git to repack first.\n>\n> I overall like the direction of this series, but I think this one is\n> just too much. While you definitely shouldn't have a ton of packs, this\n> leaves the user with no real escape hatch. And 256 isn't actually that\n> many packs.\n\nIt was raised back to 4096 at the end (I didn't know how many spare\nbits we had at this point).\n\nAgreed on the escape hatch though. I think we could do better: if\nthere are more than X packs, we repack X packs into one and leave the\nrest alone. The _next_ pack-objects will pick another X packs to\ncombine. Repeat until you only have one pack left.\n-- \nDuy\n"},{"id":"340773","messageId":"CACsJy8Bvtk_rKJ34F_PmBD5F2WN4L8ecug5dQX8-gt8i1owy3Q@mail.gmail.com","threadId":"47943","inReplyTo":"87y3jbdiof.fsf@evledraar.gmail.com","subject":"Re: [PATCH 00/11] Reduce pack-objects memory footprint","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T00:14:01Z","receivedAt":"2018-03-02T00:14:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 1, 2018 at 8:33 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted:\n>\n>> The array of object_entry in pack-objects can take a lot of memory\n>> when pack-objects is run in \"pack everything\" mode. On linux-2.6.git,\n>> this array alone takes roughly 800MB.\n>>\n>> This series reorders some fields and reduces field size... to keep\n>> this struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on\n>> 64-bit linux and saves 260MB on linux-2.6.git.\n>\n> I'm very interested in this patch series. I don't have time to test this\n> one right now (have to run), but with your previous RFC patch memory use\n> (in the ~4GB range) on a big in-house repo went down by a bit over 3%,\n> and it's ~5% faster.\n>\n> Before/after RSS 4440812 / 4290000 & runtime 172.73 / 162.45. This is\n> after having already done a full git gc before, data via /usr/bin/time\n> -v.\n\nJeff correctly pointed out elsewhere in this thread that RSS covers\nboth heap (this is what I try to reduce) and some file cache (we mmap\nthe whole pack file just to ease the reading) so RSS might not a good\nindicator of memory reduction. Any new freed memory should be used for\ncache which raises RSS back up. I think the RssAnon field in\n/proc//status shows it better.\n\n> So not huge, but respectable.\n>\n> We have a big repo, and this gets repacked on 6-8GB of memory on dev\n> KVMs, so we're under a fair bit of memory pressure. git-gc slows things\n> down a lot.\n>\n> It would be really nice to have something that made it use drastically\n> less memory at the cost of less efficient packs. Is the property that\n\nAhh.. less efficient. You may be more interested in [1] then. It\navoids rewriting the base pack. Without the base pack, book keeping\nbecomes much much cheaper.\n\nWe still read every single byte in all packs though (I think, unless\nyou use pack-bitmap) and this amount of I/O affect the rest of the\nsystem too. Perhaps reducing core.packedgitwindowsize might make it\nfriendlier to the OS, I don't know.\n\n> you need to spend give or take the size of .git/objects in memory\n> something inherent, or just a limitation of the current implementation?\n> I.e. could we do a first pass to pick some objects based on some\n> heuristic, then repack them N at a time, and finally delete the\n> now-obsolete packs?\n>\n> Another thing I've dealt with is that on these machines their\n> NFS-mounted storage gets exhausted (I'm told) due to some pathological\n> operations git does during repack, I/O tends to get 5-6x slower. Of\n> course ionice doesn't help because the local kernel doesn't know\n> anything about how harmful it is.\n\n[1] https://public-inbox.org/git/20180301092046.2769-1-pclouds@gmail.com/T/#u\n-- \nDuy\n"},{"id":"340788","messageId":"CACsJy8AzP9HLc02LbSxBvNM6Dg4FsgRt57SgaNFyT5TBPRq-Zw@mail.gmail.com","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T10:18:45Z","receivedAt":"2018-03-02T10:19:27Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 28, 2018 at 4:27 PM, Duy Nguyen wrote:\n> linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n> all apps nearly unusuable (granted the problem is partly Linux I/O\n> scheduler too). So I wonder if we can reduce pack-objects memory\n> footprint a bit.\n\nNext low hanging fruit item:\n\nstruct revindex_entry {\n off_t offset;\n unsigned int nr;\n};\n\nWe need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If\nwe break this struct apart and store two arrays of offset and nr in\nstruct packed_git, we save 4 bytes per struct, 26 MB total.\n\nIt's getting low but every megabyte counts for me, and it does not\nlook like breaking this struct will make horrible code (we recreate\nthe struct at find_pack_revindex()) so I'm going to do this too unless\nsomeone objects. There will be slight performance regression due to\ncache effects, but hopefully it's ok.\n-- \nDuy\n"},{"id":"340791","messageId":"20180302103737.GA22348@80x24.org","threadId":"47943","inReplyTo":"CACsJy8AzP9HLc02LbSxBvNM6Dg4FsgRt57SgaNFyT5TBPRq-Zw@mail.gmail.com","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2018-03-02T10:37:37Z","receivedAt":"2018-03-02T10:37:48Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Duy Nguyen wrote:\n> struct revindex_entry {\n> off_t offset;\n> unsigned int nr;\n> };\n> \n> We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If\n> we break this struct apart and store two arrays of offset and nr in\n> struct packed_git, we save 4 bytes per struct, 26 MB total.\n\nCan the offset array can be a union which stores\nint32_t/uint32_t instead of off_t for projects which never\nexceed 2/4GB?\n\nLikewise, places object_entry where \"unsigned long\" and off_t\nare 64-bit could benefit from being 32-bit. Testing/maintenance\noverhead could be bad, for those, though.\n"},{"id":"340797","messageId":"20180302105405.GD11074@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8AzP9HLc02LbSxBvNM6Dg4FsgRt57SgaNFyT5TBPRq-Zw@mail.gmail.com","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-02T10:54:06Z","receivedAt":"2018-03-02T10:54:23Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 02, 2018 at 05:18:45PM +0700, Duy Nguyen wrote:\n\n> On Wed, Feb 28, 2018 at 4:27 PM, Duy Nguyen wrote:\n> > linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n> > consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n> > all apps nearly unusuable (granted the problem is partly Linux I/O\n> > scheduler too). So I wonder if we can reduce pack-objects memory\n> > footprint a bit.\n> \n> Next low hanging fruit item:\n> \n> struct revindex_entry {\n> off_t offset;\n> unsigned int nr;\n> };\n> \n> We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If\n> we break this struct apart and store two arrays of offset and nr in\n> struct packed_git, we save 4 bytes per struct, 26 MB total.\n> \n> It's getting low but every megabyte counts for me, and it does not\n> look like breaking this struct will make horrible code (we recreate\n> the struct at find_pack_revindex()) so I'm going to do this too unless\n> someone objects. There will be slight performance regression due to\n> cache effects, but hopefully it's ok.\n\nMaybe you will prove me wrong, but I don't think splitting them is going\nto work. The point of the revindex_entry is that we sort the (offset,nr)\ntuple as a unit.\n\nOr are you planning to sort it, and then copy the result into two\nseparate arrays? I think that would work, but it sounds kind of nasty\n(arcane code, and extra CPU work for systems that don't care about the\n26MB).\n\n-Peff\n"},{"id":"340798","messageId":"CACsJy8DgNfVyp9v7NgO=1V54eBBHCToTTmxr3Xw=yrt9di8z5w@mail.gmail.com","threadId":"47943","inReplyTo":"20180302105405.GD11074@sigill.intra.peff.net","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T10:55:32Z","receivedAt":"2018-03-02T10:56:14Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 2, 2018 at 5:54 PM, Jeff King wrote:\n> On Fri, Mar 02, 2018 at 05:18:45PM +0700, Duy Nguyen wrote:\n>\n>> On Wed, Feb 28, 2018 at 4:27 PM, Duy Nguyen wrote:\n>> > linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n>> > consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n>> > all apps nearly unusuable (granted the problem is partly Linux I/O\n>> > scheduler too). So I wonder if we can reduce pack-objects memory\n>> > footprint a bit.\n>>\n>> Next low hanging fruit item:\n>>\n>> struct revindex_entry {\n>> off_t offset;\n>> unsigned int nr;\n>> };\n>>\n>> We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If\n>> we break this struct apart and store two arrays of offset and nr in\n>> struct packed_git, we save 4 bytes per struct, 26 MB total.\n>>\n>> It's getting low but every megabyte counts for me, and it does not\n>> look like breaking this struct will make horrible code (we recreate\n>> the struct at find_pack_revindex()) so I'm going to do this too unless\n>> someone objects. There will be slight performance regression due to\n>> cache effects, but hopefully it's ok.\n>\n> Maybe you will prove me wrong, but I don't think splitting them is going\n> to work. The point of the revindex_entry is that we sort the (offset,nr)\n> tuple as a unit.\n>\n> Or are you planning to sort it, and then copy the result into two\n> separate arrays?\n\nYep.\n\n> I think that would work, but it sounds kind of nasty\n\nYeah :(\n\n> (arcane code, and extra CPU work for systems that don't care about the\n> 26MB).\n-- \nDuy\n"},{"id":"340799","messageId":"20180302105710.GE11074@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8Bvtk_rKJ34F_PmBD5F2WN4L8ecug5dQX8-gt8i1owy3Q@mail.gmail.com","subject":"Re: [PATCH 00/11] Reduce pack-objects memory footprint","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-02T10:57:10Z","receivedAt":"2018-03-02T10:57:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 02, 2018 at 07:14:01AM +0700, Duy Nguyen wrote:\n\n> > We have a big repo, and this gets repacked on 6-8GB of memory on dev\n> > KVMs, so we're under a fair bit of memory pressure. git-gc slows things\n> > down a lot.\n> >\n> > It would be really nice to have something that made it use drastically\n> > less memory at the cost of less efficient packs. Is the property that\n> \n> Ahh.. less efficient. You may be more interested in [1] then. It\n> avoids rewriting the base pack. Without the base pack, book keeping\n> becomes much much cheaper.\n> \n> We still read every single byte in all packs though (I think, unless\n> you use pack-bitmap) and this amount of I/O affect the rest of the\n> system too. Perhaps reducing core.packedgitwindowsize might make it\n> friendlier to the OS, I don't know.\n\nYes, the \".keep\" thing is actually quite expensive. We still do a\ncomplete rev-list to find all the objects we want, and then for each\nobject say \"is this in a pack with .keep?\". And worse, the mru doesn't\nhelp there because even if we find it in the first pack, we have to keep\nlooking to see if it's _another_ pack.\n\nThere are probably some low-hanging optimizations there (e.g., only\nlooking in the .keep packs if that's all we're looking for; we may even\ndo that already).\n\nBut I think fundamentally you'd do much better to generate the partial\nlist of objects outside of pack-objects entirely, and then just feed it\nto pack-objects without using \"--revs\".\n\n-Peff\n"},{"id":"340807","messageId":"CACsJy8BEgC3YnFXR4EsS=cZ=q0ud9cp1NBgOvua0LmUagnKTsg@mail.gmail.com","threadId":"47943","inReplyTo":"20180302105405.GD11074@sigill.intra.peff.net","subject":"Re: Reduce pack-objects memory footprint?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-02T14:38:34Z","receivedAt":"2018-03-02T14:39:14Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 2, 2018 at 5:54 PM, Jeff King wrote:\n> On Fri, Mar 02, 2018 at 05:18:45PM +0700, Duy Nguyen wrote:\n>\n>> On Wed, Feb 28, 2018 at 4:27 PM, Duy Nguyen wrote:\n>> > linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n>> > consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n>> > all apps nearly unusuable (granted the problem is partly Linux I/O\n>> > scheduler too). So I wonder if we can reduce pack-objects memory\n>> > footprint a bit.\n>>\n>> Next low hanging fruit item:\n>>\n>> struct revindex_entry {\n>> off_t offset;\n>> unsigned int nr;\n>> };\n>>\n>> We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If\n>> we break this struct apart and store two arrays of offset and nr in\n>> struct packed_git, we save 4 bytes per struct, 26 MB total.\n>>\n>> It's getting low but every megabyte counts for me, and it does not\n>> look like breaking this struct will make horrible code (we recreate\n>> the struct at find_pack_revindex()) so I'm going to do this too unless\n>> someone objects. There will be slight performance regression due to\n>> cache effects, but hopefully it's ok.\n>\n> Maybe you will prove me wrong, but I don't think splitting them is going\n> to work. The point of the revindex_entry is that we sort the (offset,nr)\n> tuple as a unit.\n>\n> Or are you planning to sort it, and then copy the result into two\n> separate arrays? I think that would work, but it sounds kind of nasty\n> (arcane code, and extra CPU work for systems that don't care about the\n> 26MB).\n\n\nHow about two level lookups? We have\n\nstruct revindex_entry_l2 {\n uint32_t offset; /* the lowest 32 bits */\n uint32_t nr;\n};\n\nstruct revindex {\n struct revindex_entry *level1[256]; /* 8 high bits */\n};\n\nThis limits us to 1024GB pack files, which should give us some time\nbefore we have to worry about it again and most of the time we'll need\njust one or two items in level1[] so cache effects are not that bad.\nPreparing/Sorting this could be a problem though.\n-- \nDuy\n"},{"id":"340847","messageId":"20180303024706.31465-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301091052.32267-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 0/9] Reduce pack-objects memory footprint","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:46:57Z","receivedAt":"2018-03-03T02:47:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The array of object_entry in pack-objects can take a lot of memory\nwhen pack-objects is run in \"pack everything\" mode. On linux-2.6.git,\nthis array alone takes roughly 800MB.\n\nThis series reorders some fields and reduces field size... to keep\nthis struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on\n64-bit linux and saves 260MB on linux-2.6.git.\n\nv2 only differs in limits, documentation and a new escape hatch for\nthe pack file limit.\n\nNow the bad side:\n\n- the number of pack files pack-objects can handle is reduced to 16k\n (previously unlimited, v1 has it at 4k)\n- max delta chain is also limited to 4096 (previously practically\n unlimited), same as v1\n- some patches are quite invasive (e.g. replacing pointer with\n uint32_t) and reduces readability a bit.\n- it may be tricker to add more data in object_entry in the future.\n\nIn v1, if the total pack count is above the 4k limit, pack-objects\ndies. v2 is a bit smarter and only count packs that do not have the\ncompanion .keep files. This allows users with 16k+ pack files to\ncontinue to use pack-objects by first adding .keep to reduce pack\ncount, repack, remove (some) .keep files, repack again...\n\nWhile this process could be automated at least by 'git repack', given\nthe unbelievably high limit 16k, I don't think it's worth doing it.\n\nInterdiff\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\n@@ -267,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 45076f2523..55f19a1f18 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1038,7 +1038,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1061,11 +1061,27 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index) {\n+\t\tstruct packed_git *p = *found_pack;\n+\n+\t\tif (to_pack.in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t\t \"Please add .keep files to exclude\\n\"\n+\t\t\t \"some pack files and keep the number \"\n+\t\t\t \"of non-kept files below %d.\"),\n+\t\t\t 1 << OE_IN_PACK_BITS);\n+\n+\t\tp->index = to_pack.in_pack_count++;\n+\t\tto_pack.in_pack[p->index] = p;\n+\t}\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1088,6 +1104,8 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n+\t\tif (found_pack->index <= 0)\n+\t\t\tdie(\"BUG: found_pack should be NULL instead of having non-positive index\");\n \t\tentry->in_pack_idx = found_pack->index;\n \t\tentry->in_pack_offset = found_offset;\n \t}\n@@ -2978,18 +2996,12 @@ static int option_parse_unpack_unreachable(const struct option *opt,\n \n static void init_in_pack_mapping(struct packing_data *to_pack)\n {\n-\tstruct packed_git *p;\n-\tint i = 0;\n-\n \t/* let IN_PACK() return NULL if in_pack_idx is zero */\n-\tto_pack->in_pack[i++] = NULL;\n-\n-\tfor (p = packed_git; p; p = p->next, i++) {\n-\t\tif (i >= (1 << OE_IN_PACK_BITS))\n-\t\t\tdie(\"BUG: too many packs to handle!\");\n-\t\tto_pack->in_pack[i] = p;\n-\t\tp->index = i;\n-\t}\n+\tto_pack->in_pack[to_pack->in_pack_count++] = NULL;\n+\t/*\n+\t * the rest is lazily initialized only for packs that we want\n+\t * in want_object_in_pack().\n+\t */\n }\n \n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\ndiff --git a/pack-objects.h b/pack-objects.h\nindex ec4eba4ee4..a57aca5f03 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,7 +3,7 @@\n \n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n-#define OE_IN_PACK_BITS 12\n+#define OE_IN_PACK_BITS 14\n \n #define IN_PACK_POS(to_pack, obj) \\\n \t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n@@ -60,7 +60,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 96 */\n+\t/* size: 96, bit_padding: 18 bits */\n };\n \n struct packing_data {\n@@ -71,6 +71,7 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n \tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n\nNguyễn Thái Ngọc Duy (9):\n pack-objects: document holes in struct object_entry.h\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: note about in_pack_header_size\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: reorder 'hash' to pack struct object_entry\n\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 13 +-\n Documentation/git-repack.txt | 4 +-\n builtin/pack-objects.c | 207 +++++++++++++++++++----------\n cache.h | 3 +\n object.h | 1 -\n pack-bitmap-write.c | 8 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.h | 71 ++++++----\n 10 files changed, 212 insertions(+), 102 deletions(-)\n\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340848","messageId":"20180303024706.31465-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 1/9] pack-objects: document holes in struct object_entry.h","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:46:58Z","receivedAt":"2018-03-03T02:47:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..720a8e8756 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -28,6 +28,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n+\t/* XXX 28 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -40,6 +41,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n+\t/* size: 136, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340849","messageId":"20180303024706.31465-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 2/9] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:46:59Z","receivedAt":"2018-03-03T02:47:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This saves 8 bytes in sizeof(struct object_entry). On a large\nrepository like linux-2.6.git (6.5M objects), this saves us 52MB\nmemory.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 14 ++++++++++++--\n cache.h | 2 ++\n object.h | 1 -\n pack-objects.h | 8 ++++----\n 4 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..fd217cb51f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n@@ -1580,6 +1586,10 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t */\n \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n \t\t\t\t\t &entry->size);\n+\t} else {\n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->type = type;\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 720a8e8756..f8b06e2521 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -14,11 +14,11 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n+\tunsigned type:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n@@ -28,7 +28,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n-\t/* XXX 28 bits hole, try to pack */\n+\t/* XXX 22 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -41,7 +41,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n-\t/* size: 136, padding: 4 */\n+\t/* size: 128, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340850","messageId":"20180303024706.31465-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 3/9] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:00Z","receivedAt":"2018-03-03T02:47:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 33 ++++++++++++++++++++-------------\n 2 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fd217cb51f..a4dbb40824 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex f8b06e2521..6a85cc60c9 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS 2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n@@ -27,21 +42,13 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 20 bits hole, try to pack */\n \n-\t/* XXX 22 bits hole, try to pack */\n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n-\t/* size: 128, padding: 4 */\n+\n+\t/* size: 120 */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340851","messageId":"20180303024706.31465-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 4/9] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:01Z","receivedAt":"2018-03-03T02:47:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This does not give us any saving due to padding. But we will be able\nto save once we cut 4 bytes out of this struct in a subsequent patch.\n\nBecause of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 8 +++-----\n 5 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a4dbb40824..cfd97da7db 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth > (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS));\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 6a85cc60c9..2050a05a0b 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS 2\n+#define OE_DEPTH_BITS 12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -43,12 +44,9 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* XXX 20 bits hole, try to pack */\n-\n-\tint depth;\n-\n-\t/* size: 120 */\n+\t/* size: 120, bit_padding: 8 bits */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340852","messageId":"20180303024706.31465-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 5/9] pack-objects: note about in_pack_header_size","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:02Z","receivedAt":"2018-03-03T02:47:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Object header in a pack is packed really tight (see\npack-format.txt). Even with 8 bytes length, we need 9-10 bytes most,\nplus a hash (20 bytes). Which means this field only needs to store a\nnumber as big as 32 (5 bits).\n\nThis is trickier to pack tight though since a new hash algorithm is\ncoming, the number of bits needed may quickly increase. So leave it\nfor now.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 2050a05a0b..fb2a3c8f48 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -32,7 +32,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n-\tunsigned char in_pack_header_size;\n+\tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340853","messageId":"20180303024706.31465-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 6/9] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:03Z","receivedAt":"2018-03-03T02:47:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot). This saves us 8 bytes in struct object_entry.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 8 ++++++--\n 5 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex cfd97da7db..7bb5544883 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -878,7 +878,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..1360a93311 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\tIN_PACK_POS(to_pack, entry) = i;\n \n \t\tswitch (entry->type) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn IN_PACK_POS(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..f21479fe16 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = IN_PACK_POS(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex fb2a3c8f48..737e89b665 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,9 @@\n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n \n+#define IN_PACK_POS(to_pack, obj) \\\n+\t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n+\n /*\n * State flags for depth-first search used for analyzing delta cycles.\n *\n@@ -31,7 +34,6 @@ struct object_entry {\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -46,7 +48,7 @@ struct object_entry {\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 120, bit_padding: 8 bits */\n+\t/* size: 112, bit_padding: 8 bits */\n };\n \n struct packing_data {\n@@ -55,6 +57,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340854","messageId":"20180303024706.31465-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 7/9] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:04Z","receivedAt":"2018-03-03T02:47:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index isntead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 16k. For now if you hit\n16k pack files limit, pack-objects will simply fail [1].\n\nThis technically saves 7 bytes. But we don't see any of that in\npractice due to padding. The saving becomes real when we pack this\nstruct tighter later.\n\n[1] The escape hatch is .keep file to limit the non-kept pack files\n below 16k limit. Then you can go for another pack-objects run to\n combine another 16k pack files. Repeat until you're satisfied.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 +++++\n builtin/pack-objects.c | 59 +++++++++++++++++++++++-------\n cache.h | 1 +\n pack-objects.h | 18 ++++++++-\n 4 files changed, 71 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 3503c9e3e6..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7bb5544883..5818bf73ca 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -367,7 +367,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +478,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(&to_pack, entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n@@ -1024,7 +1024,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1047,11 +1047,27 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index) {\n+\t\tstruct packed_git *p = *found_pack;\n+\n+\t\tif (to_pack.in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t\t \"Please add .keep files to exclude\\n\"\n+\t\t\t \"some pack files and keep the number \"\n+\t\t\t \"of non-kept files below %d.\"),\n+\t\t\t 1 << OE_IN_PACK_BITS);\n+\n+\t\tp->index = to_pack.in_pack_count++;\n+\t\tto_pack.in_pack[p->index] = p;\n+\t}\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1074,7 +1090,9 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\tif (found_pack->index <= 0)\n+\t\t\tdie(\"BUG: found_pack should be NULL instead of having non-positive index\");\n+\t\tentry->in_pack_idx = found_pack->index;\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1417,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(&to_pack, entry)) {\n+\t\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1553,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(&to_pack, a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(&to_pack, b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1598,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(&to_pack, entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1868,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(&to_pack, trg_entry) &&\n+\t IN_PACK(&to_pack, trg_entry) == IN_PACK(&to_pack, src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -2958,6 +2978,16 @@ static int option_parse_unpack_unreachable(const struct option *opt,\n \treturn 0;\n }\n \n+static void init_in_pack_mapping(struct packing_data *to_pack)\n+{\n+\t/* let IN_PACK() return NULL if in_pack_idx is zero */\n+\tto_pack->in_pack[to_pack->in_pack_count++] = NULL;\n+\t/*\n+\t * the rest is lazily initialized only for packs that we want\n+\t * in want_object_in_pack().\n+\t */\n+}\n+\n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n@@ -3190,6 +3220,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t}\n \t}\n+\tinit_in_pack_mapping(&to_pack);\n \n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 737e89b665..83d91a0765 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,10 +3,14 @@\n \n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n+#define OE_IN_PACK_BITS 14\n \n #define IN_PACK_POS(to_pack, obj) \\\n \t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n \n+#define IN_PACK(to_pack, obj) \\\n+\t(to_pack)->in_pack[(obj)->in_pack_idx]\n+\n /*\n * State flags for depth-first search used for analyzing delta cycles.\n *\n@@ -20,10 +24,14 @@ enum dfs_state {\n \tDFS_NUM_STATES\n };\n \n+/*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -35,6 +43,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n@@ -46,9 +55,12 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 8 bits hole, try to pack */\n+\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112, bit_padding: 8 bits */\n+\t/* size: 112, padding: 4, bit_padding: 18 bits */\n };\n \n struct packing_data {\n@@ -59,6 +71,8 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340855","messageId":"20180303024706.31465-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 8/9] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:05Z","receivedAt":"2018-03-03T02:48:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Notice that packing_data::nr_objects is uint32_t, we could only handle\nmaximum 4G objects and can address all of them with an uint32_t. If we\nuse a pointer here, we waste 4 bytes on 64 bit architecture.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\nThere are holes in this struct but this patch is already big. Struct\npacking can be done separately. Even with holes, we save 8 bytes per\nobject_entry.\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 124 +++++++++++++++++++++++------------------\n pack-objects.h | 14 +++--\n 2 files changed, 78 insertions(+), 60 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5818bf73ca..55f19a1f18 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,20 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define DELTA(obj) \\\n+\t((obj)->delta_idx ? &to_pack.objects[(obj)->delta_idx - 1] : NULL)\n+#define DELTA_CHILD(obj) \\\n+\t((obj)->delta_child_idx ? &to_pack.objects[(obj)->delta_child_idx - 1] : NULL)\n+#define DELTA_SIBLING(obj) \\\n+\t((obj)->delta_sibling_idx ? &to_pack.objects[(obj)->delta_sibling_idx - 1] : NULL)\n+\n+#define CLEAR_DELTA(obj) (obj)->delta_idx = 0\n+#define CLEAR_DELTA_CHILD(obj) (obj)->delta_child_idx = 0\n+#define CLEAR_DELTA_SIBLING(obj) (obj)->delta_sibling_idx = 0\n+\n+#define SET_DELTA(obj, val) (obj)->delta_idx = ((val) - to_pack.objects) + 1\n+#define SET_DELTA_CHILD(obj, val) (obj)->delta_child_idx = ((val) - to_pack.objects) + 1\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -125,11 +139,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -286,12 +300,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -315,7 +329,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -341,7 +355,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -377,8 +391,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -406,7 +420,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -425,7 +439,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -465,13 +479,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -486,7 +500,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (entry->type != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -538,12 +552,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tCLEAR_DELTA(e);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -605,34 +619,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -643,7 +657,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -658,8 +672,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tCLEAR_DELTA_CHILD(&objects[i]);\n+\t\tCLEAR_DELTA_SIBLING(&objects[i]);\n \t}\n \n \t/*\n@@ -669,11 +683,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1509,10 +1523,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\tentry->type = entry->in_pack_type;\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1583,17 +1597,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tCLEAR_DELTA(entry);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1635,7 +1651,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1660,7 +1676,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1683,7 +1699,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1698,7 +1714,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1881,7 +1897,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1957,7 +1973,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1986,7 +2002,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1995,13 +2011,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2070,7 +2086,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2120,7 +2136,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2128,7 +2144,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2449,7 +2465,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 83d91a0765..db50e56223 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -33,11 +33,13 @@ struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n+\t/* XXX 4 bytes hole, try to pack */\n+\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -60,7 +62,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112, padding: 4, bit_padding: 18 bits */\n+\t/* size: 104, padding: 4, bit_padding: 18 bits */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"340856","messageId":"20180303024706.31465-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v2 9/9] pack-objects: reorder 'hash' to pack struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-03T02:47:06Z","receivedAt":"2018-03-03T02:48:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex db50e56223..a57aca5f03 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -38,12 +38,10 @@ struct object_entry {\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\t/* XXX 4 bytes hole, try to pack */\n-\n+\tuint32_t hash;\t\t\t/* name hint hash */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -62,7 +60,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 104, padding: 4, bit_padding: 18 bits */\n+\t/* size: 96, bit_padding: 18 bits */\n };\n \n struct packing_data {\n-- \n2.16.1.435.g8f24da2e1a\n\n"},{"id":"341000","messageId":"20180305092842.GA28090@ash","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"Re: [PATCH/RFC v2 0/9] Reduce pack-objects memory footprint","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-05T09:28:42Z","receivedAt":"2018-03-05T09:29:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 03, 2018 at 09:46:57AM +0700, Nguyễn Thái Ngọc Duy wrote:\n> The array of object_entry in pack-objects can take a lot of memory\n> when pack-objects is run in \"pack everything\" mode. On linux-2.6.git,\n> this array alone takes roughly 800MB.\n> \n> This series reorders some fields and reduces field size... to keep\n> this struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on\n> 64-bit linux and saves 260MB on linux-2.6.git.\n\nAnd I continue to push this until someone screams \"enough is enough!\".\nThis patch saves 4 more bytes. The trade off is, processing objects\nwith offset beyond 4 GB will be slower. But I think this is a reasonable\ntrade off.\n\nThe same trick could be done for \"size\" field in this struct\n(i.e. uncompressed object size greater than 32 bits must be read back\nfrom disk). Interestingly though, \"size\" is unsigned long which is 32\nbits on Windows and nobody has complained about it so far, we could\neven just unconditionally reject objects larger than 4GB.\n\n-- 8< --\nSubject: [PATCH] pack-objects: shrink in_pack_offset for 4GB pack files\n\nIf a pack file is smaller than 4GB, pack offsets should fit within 32\nbits. If not (which is not considered a common case), this field\nin_pack_location stores the object index instead (which still fits in 32\nbits) and getting pack offset requires extra lookups through\nnth_packed_object_offset() function call.\n\nThis saves us 4 bytes but lose it to padding until this struct is shrunk\nfurther.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 26 ++++++++++++++-----------\n pack-objects.h | 44 ++++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 57 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 55f19a1f18..57c04b277b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -43,6 +43,8 @@\n #define SET_DELTA(obj, val) (obj)->delta_idx = ((val) - to_pack.objects) + 1\n #define SET_DELTA_CHILD(obj, val) (obj)->delta_child_idx = ((val) - to_pack.objects) + 1\n \n+#define IN_PACK_OFFSET(obj) oe_in_pack_offset(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -397,7 +399,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n \n-\toffset = entry->in_pack_offset;\n+\toffset = IN_PACK_OFFSET(entry);\n \trevidx = find_pack_revindex(p, offset);\n \tdatalen = revidx[1].offset - offset;\n \tif (!pack_to_stdout && p->index_version > 1 &&\n@@ -1107,7 +1109,7 @@ static void create_object_entry(const struct object_id *oid,\n \t\tif (found_pack->index <= 0)\n \t\t\tdie(\"BUG: found_pack should be NULL instead of having non-positive index\");\n \t\tentry->in_pack_idx = found_pack->index;\n-\t\tentry->in_pack_offset = found_offset;\n+\t\toe_set_in_pack_offset(&to_pack, entry, found_offset);\n \t}\n \n \tentry->no_try_delta = no_try_delta;\n@@ -1442,7 +1444,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned char *buf, c;\n \t\tenum object_type type;\n \n-\t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n+\t\tbuf = use_pack(p, &w_curs, IN_PACK_OFFSET(entry), &avail);\n \n \t\t/*\n \t\t * We want in_pack_type even if we do not reuse delta\n@@ -1475,12 +1477,12 @@ static void check_object(struct object_entry *entry)\n \t\tcase OBJ_REF_DELTA:\n \t\t\tif (reuse_delta && !entry->preferred_base)\n \t\t\t\tbase_ref = use_pack(p, &w_curs,\n-\t\t\t\t\t\tentry->in_pack_offset + used, NULL);\n+\t\t\t\t\t\tIN_PACK_OFFSET(entry) + used, NULL);\n \t\t\tentry->in_pack_header_size = used + 20;\n \t\t\tbreak;\n \t\tcase OBJ_OFS_DELTA:\n \t\t\tbuf = use_pack(p, &w_curs,\n-\t\t\t\t entry->in_pack_offset + used, NULL);\n+\t\t\t\t IN_PACK_OFFSET(entry) + used, NULL);\n \t\t\tused_0 = 0;\n \t\t\tc = buf[used_0++];\n \t\t\tofs = c & 127;\n@@ -1494,8 +1496,8 @@ static void check_object(struct object_entry *entry)\n \t\t\t\tc = buf[used_0++];\n \t\t\t\tofs = (ofs << 7) + (c & 127);\n \t\t\t}\n-\t\t\tofs = entry->in_pack_offset - ofs;\n-\t\t\tif (ofs <= 0 || ofs >= entry->in_pack_offset) {\n+\t\t\tofs = IN_PACK_OFFSET(entry) - ofs;\n+\t\t\tif (ofs <= 0 || ofs >= IN_PACK_OFFSET(entry)) {\n \t\t\t\terror(\"delta base offset out of bound for %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t\t\tgoto give_up;\n@@ -1538,7 +1540,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * object size from the delta header.\n \t\t\t */\n \t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n+\t\t\t\tIN_PACK_OFFSET(entry) + entry->in_pack_header_size);\n \t\t\tif (entry->size == 0)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n@@ -1578,8 +1580,8 @@ static int pack_offset_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n-\treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n-\t\t\t(a->in_pack_offset > b->in_pack_offset);\n+\treturn IN_PACK_OFFSET(a) < IN_PACK_OFFSET(b) ? -1 :\n+\t\t(IN_PACK_OFFSET(a) > IN_PACK_OFFSET(b));\n }\n \n /*\n@@ -1614,7 +1616,9 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(IN_PACK(&to_pack, entry), entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(&to_pack, entry),\n+\t\t\t IN_PACK_OFFSET(entry),\n+\t\t\t &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\ndiff --git a/pack-objects.h b/pack-objects.h\nindex a57aca5f03..cb752fb4d8 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#include \"packfile.h\"\n+\n #define OE_DFS_STATE_BITS 2\n #define OE_DEPTH_BITS 12\n #define OE_IN_PACK_BITS 14\n@@ -32,7 +34,6 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n@@ -42,6 +43,13 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\t/*\n+\t * If in_pack_location_is_offset is true, this contains offset\n+\t * to the object in \"in_pack\". If false, it contains the\n+\t * object _index_ and pack offset must be retrieved via\n+\t * nth_packed_object_offset().\n+\t */\n+\tuint32_t in_pack_location;\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -54,13 +62,14 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned in_pack_location_is_offset:1;\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n \t/* XXX 8 bits hole, try to pack */\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 96, bit_padding: 18 bits */\n+\t/* size: 96, padding: 4, bit_padding: 17 bits */\n };\n \n struct packing_data {\n@@ -103,4 +112,35 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline off_t oe_in_pack_offset(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->in_pack_location_is_offset)\n+\t\treturn e->in_pack_location;\n+\n+\t/*\n+\t * Slow path where in_pack_location contains the object index\n+\t * instead of offset. We perform one more lookup.\n+\t */\n+\treturn nth_packed_object_offset(IN_PACK(pack, e),\n+\t\t\t\t\te->in_pack_location);\n+}\n+\n+static inline void oe_set_in_pack_offset(struct packing_data *pack,\n+\t\t\t\t\t struct object_entry *e,\n+\t\t\t\t\t off_t offset)\n+{\n+\tstruct revindex_entry *revidx;\n+\n+\te->in_pack_location = offset;\n+\tif (e->in_pack_location == offset) {\n+\t\te->in_pack_location_is_offset = 1;\n+\t\treturn;\n+\t}\n+\n+\trevidx = find_pack_revindex(IN_PACK(pack, e), offset);\n+\te->in_pack_location = revidx->nr;\n+\te->in_pack_location_is_offset = 0;\n+}\n+\n #endif\n-- \n2.16.1.435.g8f24da2e1a\n\n-- 8< --\n"},{"id":"341019","messageId":"87bmg2oc5r.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180301092046.2769-2-pclouds@gmail.com","subject":"Re: [PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-05T14:00:16Z","receivedAt":"2018-03-05T14:00:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> pack-objects could be a big memory hog especially on large repos,\n> everybody knows that. The suggestion to stick a .keep file on the\n> largest pack to avoid this problem is also known for a long time.\n>\n> Let's do the suggestion automatically instead of waiting for people to\n> come to Git mailing list and get the advice. When a certain condition\n> is met, gc --auto create a .keep file temporary before repack is run,\n> then remove it afterward.\n>\n> gc --auto does this based on an estimation of pack-objects memory\n> usage and whether that fits in one third of system memory (the\n> assumption here is for desktop environment where there are many other\n> applications running).\n>\n> Since the estimation may be inaccurate and that 1/3 threshold is\n> arbitrary, give the user a finer control over this mechanism as well:\n> if the largest pack is larger than gc.bigPackThreshold, it's kept.\n\nThis is very promising. Saves lots of memory on my ad-hoc testing of\nadding a *.keep file on an in-house repo.\n\n> +\tif (big_pack_threshold)\n> +\t\treturn pack->pack_size >= big_pack_threshold;\n> +\n> +\t/* First we have to scan through at least one pack */\n> +\tmem_want = pack->pack_size + pack->index_size;\n> +\t/* then pack-objects needs lots more for book keeping */\n> +\tmem_want += sizeof(struct object_entry) * nr_objects;\n> +\t/*\n> +\t * internal rev-list --all --objects takes up some memory too,\n> +\t * let's say half of it is for blobs\n> +\t */\n> +\tmem_want += sizeof(struct blob) * nr_objects / 2;\n> +\t/*\n> +\t * and the other half is for trees (commits and tags are\n> +\t * usually insignificant)\n> +\t */\n> +\tmem_want += sizeof(struct tree) * nr_objects / 2;\n> +\t/* and then obj_hash[], underestimated in fact */\n> +\tmem_want += sizeof(struct object *) * nr_objects;\n> +\t/*\n> +\t * read_sha1_file() (either at delta calculation phase, or\n> +\t * writing phase) also fills up the delta base cache\n> +\t */\n> +\tmem_want += delta_base_cache_limit;\n> +\t/* and of course pack-objects has its own delta cache */\n> +\tmem_want += max_delta_cache_size;\n\nI'm not familiar enough with this part to say, but isn't this assuming a\nlot about the distribution of objects in a way that will cause is not to\nrepack in some pathological cases?\n\nProbably worth documenting...\n\n> +\t/* Only allow 1/3 of memory for pack-objects */\n> +\tmem_have = total_ram() / 3;\n\nWould be great to have this be a configurable variable, so you could set\nit to e.g. 33% (like here), 50% etc.\n"},{"id":"341118","messageId":"20180306104158.6541-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180301092046.2769-1-pclouds@gmail.com","subject":"[PATCH v2 0/5] Avoid expensive 'repack -ad' in gc --auto","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:53Z","receivedAt":"2018-03-06T10:42:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"I'm pretty happy with this now. Like v1, this is about not touching\nthe giant base pack when doing background gc. This saves about 2/3 of\nmemory, which in turn should improve performance if you're under\nmemory pressure.\n\nv2 changes:\n\n- the core idea remains the same, mem_have is increased to half total\n memory though. I figure including the whole mmap'd pack in the\n memory estimation may be a bit much, which is why I make this\n change.\n- no creating .keep files temporarily\n- the config key is renamed gc.bigBasePackThreshold (named after\n core.bigFileThreshold)\n- note that if you set gc.bigFileThreshold, then normal gc (without\n --auto) can trigger this mode too.\n- you can also control this with --[no-]keep-base-pack\n- documents and tests\n- some more progress output improvements\n\nI'm _not_ doing external rev-list in this series though. I found out\nthat we have added more and more stuff in the internal rev-list code\npath over the year and simply running\n\n git rev-list .... | git pack-objects\n\nwill break stuff (the test suite first for example). I will do it\nbecause it does help. But it will take some time.\n\nPS. This conflicts with sb/packfiles-in-repository on 'pu' because I\nintroduced new references to the global variable \"packed_git\" and\nprepare_packed_git(). Resolving this should be simple though:\n\n- drop prepare_packed_git()\n- replace packed_git with get_packed_git(the_repository)\n\nNguyễn Thái Ngọc Duy (5):\n fixup! Add a test showing that 'git repack' throws away grafted-away\n parents\n repack: add --keep-pack option\n gc --auto: exclude base pack if not enough mem to \"repack -ad\"\n pack-objects: show some progress when counting kept objects\n pack-objects: display progress in get_object_details()\n\n Documentation/config.txt | 7 ++\n Documentation/git-gc.txt | 13 +++\n Documentation/git-pack-objects.txt | 4 +\n Documentation/git-repack.txt | 4 +\n builtin/gc.c | 153 +++++++++++++++++++++++++++--\n builtin/pack-objects.c | 51 ++++++++--\n builtin/repack.c | 23 ++++-\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 29 ++++++\n t/t7700-repack.sh | 21 +++-\n 12 files changed, 295 insertions(+), 17 deletions(-)\n\n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341119","messageId":"20180306104158.6541-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v2 1/5] fixup! Add a test showing that 'git repack' throws away grafted-away parents","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:54Z","receivedAt":"2018-03-06T10:42:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The closing quote of a test body by convention is always at the start\nof line.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t7700-repack.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 6061a04147..38247afbec 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -194,7 +194,7 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&\n \tgit repack -a -d &&\n \tgit cat-file -t $H1\n-\t'\n+'\n \n test_done\n \n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341120","messageId":"20180306104158.6541-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v2 2/5] repack: add --keep-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:55Z","receivedAt":"2018-03-06T10:42:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We allow to keep existing packs by having companion .keep files. This\nis helpful when a pack is permanently kept. In the next patch, git-gc\njust wants to keep a pack temporarily, for one pack-objects\nrun. git-gc can use --keep-pack for this use case.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 4 ++++\n Documentation/git-repack.txt | 4 ++++\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++++++++\n builtin/repack.c | 23 +++++++++++++++++++---\n t/t7700-repack.sh | 19 ++++++++++++++++++\n 5 files changed, 78 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..1975477160 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -126,6 +126,10 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n+--keep-pack=::\n+\tIgnore the given pack. This is the equivalent of having\n+\t`.keep` file on the pack. Implies `--honor-pack-keep`.\n+\n --incremental::\n \tThis flag causes an object already in a pack to be ignored\n \teven if it would have otherwise been packed.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..12b073e115 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -133,6 +133,10 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--keep-pack=::\n+\tExclude the given pack from repacking. This is the equivalent\n+\tof having `.keep` file on the pack. Implies `--pack-kept-objects`.\n+\n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\n \tobjects older than ``. This can be used to optimize out\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..8e3f870d71 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -28,6 +28,7 @@\n #include \"argv-array.h\"\n #include \"list.h\"\n #include \"packfile.h\"\n+#include \"dir.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -2917,6 +2918,32 @@ static void get_object_list(int ac, const char **av)\n \toid_array_clear(&recent_objects);\n }\n \n+static void add_extra_kept_packs(const struct string_list *names)\n+{\n+\tstruct packed_git *p;\n+\n+\tif (!names->nr)\n+\t\treturn;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tconst char *name = basename(p->pack_name);\n+\t\tint i;\n+\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < names->nr; i++) {\n+\t\t\tif (fspathcmp(name, names->items[i].string))\n+\t\t\t\tcontinue;\n+\n+\t\t\tp->pack_keep = 1;\n+\t\t\tignore_packed_keep = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n static int option_parse_index_version(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -2956,6 +2983,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t N_(\"do not show progress meter\"), 0),\n@@ -3022,6 +3050,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n \t\t\t N_(\"pack compression level\")),\n \t\tOPT_SET_INT(0, \"keep-true-parents\", &grafts_replace_parents,\n@@ -3150,6 +3180,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tprogress = 2;\n \n \tprepare_packed_git();\n+\tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep) {\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..6a1dade0e1 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n * have a corresponding .keep or .promisor file. These packs are not to\n * be kept if we are going to pack everything into one file.\n */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list)\n+static void get_non_kept_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t\tconst struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \twhile ((e = readdir(dir)) != NULL) {\n \t\tsize_t len;\n+\t\tint i;\n+\n+\t\tfor (i = 0;i < extra_keep->nr; i++)\n+\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n+\t\t\t\tbreak;\n+\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n+\t\t\tcontinue;\n+\n \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n \t\t\tcontinue;\n \n@@ -148,7 +157,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tint ext, ret, failed;\n+\tint i, ext, ret, failed;\n \tFILE *out;\n \n \t/* variables to be filled by option parsing */\n@@ -160,6 +169,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *depth = NULL;\n \tconst char *threads = NULL;\n \tconst char *max_pack_size = NULL;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_reuse_delta = 0, no_reuse_object = 0;\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n@@ -200,6 +210,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"maximum size of each packfile\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -215,6 +227,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))\n \t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n \n+\tif (keep_pack_list.nr && pack_kept_objects > 0)\n+\t\tdie(_(\"incompatible --keep-pack and --pack-kept-objects\"));\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps;\n \n@@ -230,6 +244,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--keep-true-parents\");\n \tif (!pack_kept_objects)\n \t\targv_array_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list.nr; i++)\n+\t\targv_array_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t\t keep_pack_list.items[i].string);\n \targv_array_push(&cmd.args, \"--non-empty\");\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n@@ -254,7 +271,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&cmd.args, \"--write-bitmap-index\");\n \n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs);\n+\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n \t\t\tif (unpack_unreachable) {\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 38247afbec..553d907d34 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -196,5 +196,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit cat-file -t $H1\n '\n \n+test_expect_success 'repack --keep-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tfor cmit in one two three four; do\n+\t\t\ttest_commit $cmit &&\n+\t\t\tgit repack -d\n+\t\tdone &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 4 pack-list &&\n+\t\tKEEP1=`head -n1 pack-list` &&\n+\t\tKEEP4=`tail -n1 pack-list` &&\n+\t\tgit repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n+\t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\ttest_line_count = 3 new-counts &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n \n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341121","messageId":"20180306104158.6541-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:56Z","receivedAt":"2018-03-06T10:42:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\ngiant base pack to avoid this problem is also known for a long time.\n\nLet's do the suggestion automatically instead of waiting for people to\ncome to Git mailing list and get the advice. When a certain condition\nis met, \"gc --auto\" tells \"git repack\" to keep the base pack around.\nThe end result would be two packs instead of one.\n\nOn linux-2.6.git, valgrind massif reports 1.6GB heap in \"pack all\"\ncase, and 535MB [1] in \"pack all except the base pack\" case. We save\nroughly 1GB memory by excluding the base pack.\n\ngc --auto decides to do this based on an estimation of pack-objects\nmemory usage, which is quite accurate at least for the heap part, and\nwhether that fits in half of system memory (the assumption here is for\ndesktop environment where there are many other applications running).\n\nSince the estimation may be inaccurate and that 1/2 threshold is\nreally arbitrary, give the user a finer control over this mechanism:\nif the largest pack is larger than gc.bigBasePackThreshold, it's kept.\n\nPS. A big chunk of the remaining 535MB is the result of pack-objects\nrunning rev-list internally. This will be dealt with when we could run\nrev-list externally. Right now we can't because pack-objects internal\nrev-list does more regarding unreachable objects, which cannot be done\nby \"git rev-list\".\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 7 ++\n Documentation/git-gc.txt | 13 ++++\n builtin/gc.c | 153 +++++++++++++++++++++++++++++++++++++--\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 29 ++++++++\n 8 files changed, 204 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..120cf6bac9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1549,6 +1549,13 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.bigBasePackThreshold::\n+\tMake `git gc --auto` only enable `--keep-base-pack` when the\n+\tbase pack's size is larger than this limit (in bytes).\n+\tDefaults to zero, which disables this check and lets\n+\t`git gc --auto` determine when to enable `--keep-base-pack`\n+\tbased on memory usage.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..35ad420d5c 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -59,6 +59,11 @@ then existing packs (except those marked with a `.keep` file)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n automatic consolidation of packs.\n++\n+If the physical amount of memory is considered not enough for `git\n+repack` to run smoothly, `--keep-base-pack` is enabled. This could be\n+overridden by setting `gc.bigBasePackThreshold` which only enables\n+`--keep-base-pack` when the base pack is larger the specified limit.\n \n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\n@@ -78,6 +83,10 @@ automatic consolidation of packs.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n+--keep-base-pack::\n+\tAll packs except the base pack are consolidated into a single\n+\tpack. The largest pack is considered the base pack.\n+\n Configuration\n -------------\n \n@@ -167,6 +176,10 @@ run commands concurrently have to live with some risk of corruption (which\n seems to be low in practice) unless they turn off automatic garbage\n collection with 'git config gc.auto 0'.\n \n+Set environment variable `GIT_TRACE` in order to see the memory usage\n+estimation in `git gc --auto` that determines whether the base pack is\n+kept.\n+\n HOOKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..273657ddf4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,10 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -39,6 +43,8 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_base_pack_threshold;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -126,6 +132,9 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigbasepackthreshold\", &big_base_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -164,6 +173,21 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n+static struct packed_git *find_the_base_pack(void)\n+{\n+\tstruct packed_git *p, *base = NULL;\n+\n+\tprepare_packed_git();\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_local &&\n+\t\t (!base || base->pack_size < p->pack_size))\n+\t\t\tbase = p;\n+\t}\n+\n+\treturn base;\n+}\n+\n static int too_many_packs(void)\n {\n \tstruct packed_git *p;\n@@ -187,7 +211,101 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static inline unsigned long total_ram(void)\n+{\n+\tunsigned long default_ram = 4;\n+#ifdef HAVE_SYSINFO\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tsize_t length;\n+\n+\tmib[0] = CTL_HW;\n+\tmib[1] = HW_MEMSIZE;\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo;ullTotalPhys;\n+#else\n+\tfprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n+\t\tdefault_ram);\n+#endif\n+\treturn default_ram * 1024 * 1024 * 1024;\n+}\n+\n+static int pack_objects_uses_too_much_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t mem_want, mem_have, os_cache, heap;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\tif (big_base_pack_threshold)\n+\t\treturn pack->pack_size >= big_base_pack_threshold;\n+\n+\t/*\n+\t * First we have to scan through at least one pack.\n+\t * Assume enough room in OS file cache to keep the entire pack\n+\t * or we may accidentally evict data of other processes from\n+\t * the cache.\n+\t */\n+\tos_cache = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\theap = sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\theap += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\theap += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\theap += sizeof(struct object *) * nr_objects;\n+\t/* revindex is used also */\n+\theap += sizeof(struct revindex_entry) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\theap += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\theap += max_delta_cache_size;\n+\n+\t/*\n+\t * Only allow 1/2 of memory for pack-objects, leave the rest\n+\t * for the OS and other processes in the system.\n+\t */\n+\tmem_have = total_ram() / 2;\n+\tmem_want = os_cache + heap;\n+\n+\ttrace_printf(\"gc mem estimation\\n\"\n+\t\t \"mem_have: %\" PRIuMAX \", mem_want: %\" PRIuMAX \", \"\n+\t\t \"heap: %\" PRIuMAX \"\\n\"\n+\t\t \"pack_size: %\" PRIuMAX \", index_size: %\" PRIuMAX \", \"\n+\t\t \"nr_objects: %\" PRIuMAX \"\\n\"\n+\t\t \"base_cache: %\" PRIuMAX \", delta_cache: %\" PRIuMAX \"\\n\",\n+\t\t (uintmax_t)mem_have, (uintmax_t)mem_want, (uintmax_t)heap,\n+\t\t (uintmax_t)pack->pack_size, (uintmax_t)pack->index_size,\n+\t\t (uintmax_t)nr_objects,\n+\t\t (uintmax_t)delta_base_cache_limit, (uintmax_t)max_delta_cache_size);\n+\n+\treturn mem_want >= mem_have;\n+}\n+\n+static void add_repack_all_option(struct packed_git *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -196,6 +314,10 @@ static void add_repack_all_option(void)\n \t\tif (prune_expire)\n \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n \t}\n+\n+\tif (keep_pack)\n+\t\targv_array_pushf(&repack, \"--keep-pack=%s\",\n+\t\t\t\t basename(keep_pack->pack_name));\n }\n \n static void add_repack_incremental_option(void)\n@@ -218,9 +340,14 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option();\n-\telse if (too_many_loose_objects())\n+\tif (too_many_packs()) {\n+\t\tstruct packed_git *exclude = find_the_base_pack();\n+\n+\t\tif (!pack_objects_uses_too_much_memory(exclude))\n+\t\t\texclude = NULL;\n+\n+\t\tadd_repack_all_option(exclude);\n+\t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -353,6 +480,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tconst char *name;\n \tpid_t pid;\n \tint daemonized = 0;\n+\tint keep_base_pack = -1;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -362,6 +490,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n+\t\tOPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the base pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -427,8 +557,19 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t */\n \t\t\tdaemonized = !daemonize();\n \t\t}\n-\t} else\n-\t\tadd_repack_all_option();\n+\t} else {\n+\t\tstruct packed_git *base_pack = find_the_base_pack();\n+\t\tstruct packed_git *exclude = NULL;\n+\n+\t\tif (keep_base_pack != -1) {\n+\t\t\tif (keep_base_pack)\n+\t\t\t\texclude = base_pack;\n+\t\t} else if (base_pack && big_base_pack_threshold &&\n+\t\t\t base_pack->pack_size >= big_base_pack_threshold)\n+\t\t\texclude = base_pack;\n+\n+\t\tadd_repack_all_option(exclude);\n+\t}\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8e3f870d71..fcdd398eb7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -79,7 +79,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 68b2ad531e..a84b21986d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 41b0be575d..863fdbb0fd 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -5,6 +5,13 @@ test_description='basic git gc tests\n \n . ./test-lib.sh\n \n+test_expect_success 'setup' '\n+\t# do not let the amount of physical memory affects gc\n+\t# behavior, make sure the pack_objects_uses_too_much_memory()\n+\t# always returns false\n+\tgit config gc.bigBasePackThreshold 2g\n+'\n+\n test_expect_success 'gc empty repository' '\n \tgit gc\n '\n@@ -116,6 +123,28 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_path_is_file .git/refs/heads/should-be-loose\n '\n \n+test_expect_success 'gc --keep-base-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tfor i in 10; do\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\tfor i in 10; do\n+\t\t\ttest_commit more-$i\n+\t\tdone &&\n+\t\tgit gc --keep-base-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341122","messageId":"20180306104158.6541-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v2 4/5] pack-objects: show some progress when counting kept objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:57Z","receivedAt":"2018-03-06T10:42:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only show progress when there are new objects to be packed. But\nwhen --keep-pack is specified on the base pack, we will exclude most\nof objects. This makes 'pack-objects' stay silent for a long time\nwhile the counting phase is going.\n\nLet's show some progress whenever we visit an object instead. The\nnumber of packed objects will be shown after if it's not the same as\nthe number of visited objects.\n\nSince the meaning of this number has changed, use another word instead\nof \"Counting\" to hint about the change.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fcdd398eb7..24af4068a9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -44,7 +44,7 @@ static const char *pack_usage[] = {\n static struct packing_data to_pack;\n \n static struct pack_idx_entry **written_list;\n-static uint32_t nr_result, nr_written;\n+static uint32_t nr_result, nr_written, nr_seen;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n@@ -1092,6 +1092,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, nr_seen++);\n+\n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n \n@@ -1107,8 +1109,6 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tcreate_object_entry(oid, type, pack_name_hash(name),\n \t\t\t exclude, name && no_try_delta(name),\n \t\t\t index_pos, found_pack, found_offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1119,6 +1119,8 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, nr_seen++);\n+\n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n \n@@ -1126,8 +1128,6 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, index_pos, pack, offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -3205,7 +3205,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n+\t\tprogress_state = start_progress(_(\"Enumerating objects\"), 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341123","messageId":"20180306104158.6541-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v2 5/5] pack-objects: display progress in get_object_details()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-06T10:41:58Z","receivedAt":"2018-03-06T10:42:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This code is mostly about reading object headers, which is cheap. But\nwhen the number of objects is very large (e.g. 6.5M on linux-2.6.git)\nand the system is under memory pressure, this could take some time (86\nseconds on my system).\n\nShow something during this time to let the user know pack-objects is\nstill going strong.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 24af4068a9..2ec911bf10 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1713,6 +1713,10 @@ static void get_object_details(void)\n \tuint32_t i;\n \tstruct object_entry **sorted_by_offset;\n \n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Getting object details\"),\n+\t\t\t\t\t\tto_pack.nr_objects);\n+\n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n \tfor (i = 0; i < to_pack.nr_objects; i++)\n \t\tsorted_by_offset[i] = to_pack.objects + i;\n@@ -1723,7 +1727,9 @@ static void get_object_details(void)\n \t\tcheck_object(entry);\n \t\tif (big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n+\t\tdisplay_progress(progress_state, i + 1);\n \t}\n+\tstop_progress(&progress_state);\n \n \t/*\n \t * This must happen in a second pass, since we rely on the delta\n-- \n2.16.2.784.gb291bd247e\n\n"},{"id":"341135","messageId":"xmqqsh9d5c28.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"Re: [PATCH v2 0/5] Avoid expensive 'repack -ad' in gc --auto","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-06T17:49:35Z","receivedAt":"2018-03-06T17:49:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> I'm pretty happy with this now. Like v1, this is about not touching\n> the giant base pack when doing background gc. This saves about 2/3 of\n> memory, which in turn should improve performance if you're under\n> memory pressure.\n\nThanks. I've quickly scanned them and the overall idea looks quite\nsound. I'll comment on individual changes if needed.\n\nWill queue.\n\n"},{"id":"341136","messageId":"xmqqo9k15bhs.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180306104158.6541-2-pclouds@gmail.com","subject":"Re: [PATCH v2 1/5] fixup! Add a test showing that 'git repack' throws away grafted-away parents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-06T18:01:51Z","receivedAt":"2018-03-06T18:02:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> The closing quote of a test body by convention is always at the start\n> of line.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\nObviously correct. While the title may technically be correct, the\noriginal commit is age old that we won't be amending it anyway, and\nit is not very helpful not to say which aspect of the original is\nbeing fixed up. Perhaps something like:\n\n t7700: have closing quote of a test at the beginning of line\n\n 1ec64827 (\"Add a test showing that 'git repack' throws away\n grafted-away parents\", 2009-07-23) added this test but indented\n the closing quote by mistake.\n\nif we really cared documenting where the blame lies, but I do not\nthink it is quite worth it; anybody who cares deeply can ask \"git\nblame\" about it, so I'll just retitle and use your original log\nmessage body.\n\nThanks.\n\n> t/t7700-repack.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index 6061a04147..38247afbec 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -194,7 +194,7 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n> \tgit reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&\n> \tgit repack -a -d &&\n> \tgit cat-file -t $H1\n> -\t'\n> +'\n> \n> test_done\n"},{"id":"341139","messageId":"xmqq1sgx5adl.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180306104158.6541-3-pclouds@gmail.com","subject":"Re: [PATCH v2 2/5] repack: add --keep-pack option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-06T18:25:58Z","receivedAt":"2018-03-06T18:26:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> +--keep-pack=::\n> +\tIgnore the given pack. This is the equivalent of having\n> +\t`.keep` file on the pack. Implies `--honor-pack-keep`.\n> +\n\nA few questions I am not sure how I would answer:\n\n - Do we want to have this listed in the SYNOPSIS section, too?\n\n - We would want to make the SP in \"\" consistent with\n the dash in \"\" in the same document; which way do\n we make it uniform?\n\n - Is this description clear enough to convey that we allow more\n than one instance of this option specified, and the pack names\n accumulate?\n\n - Are there use cases where we want to _ignore_ on-disk \".keep\" and\n only honor the ones given via the \"--keep-pack\" options?\n\n - Is this description clear enough to convey that is\n just the filename part (i.e. \"pack-[0-9a-f]{40}.pack\") in our\n local $GIT_OBJECT_DIRECTORY/pack/ and not a full path to the\n packfile? I think that design is sensible, simplifies the\n implementation and reduces mistakes.\n\n> +static void add_extra_kept_packs(const struct string_list *names)\n> +{\n> +\tstruct packed_git *p;\n> +\n> +\tif (!names->nr)\n> +\t\treturn;\n> +\n> +\tprepare_packed_git();\n> +\tfor (p = packed_git; p; p = p->next) {\n> +\t\tconst char *name = basename(p->pack_name);\n> +\t\tint i;\n> +\n> +\t\tif (!p->pack_local)\n> +\t\t\tcontinue;\n> +\n> +\t\tfor (i = 0; i < names->nr; i++) {\n> +\t\t\tif (fspathcmp(name, names->items[i].string))\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tp->pack_keep = 1;\n> +\t\t\tignore_packed_keep = 1;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +}\n\nOK.\n\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 7bdb40142f..6a1dade0e1 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n> * have a corresponding .keep or .promisor file. These packs are not to\n> * be kept if we are going to pack everything into one file.\n> */\n> -static void get_non_kept_pack_filenames(struct string_list *fname_list)\n> +static void get_non_kept_pack_filenames(struct string_list *fname_list,\n> +\t\t\t\t\tconst struct string_list *extra_keep)\n> {\n> \tDIR *dir;\n> \tstruct dirent *e;\n> @@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n> \n> \twhile ((e = readdir(dir)) != NULL) {\n> \t\tsize_t len;\n> +\t\tint i;\n> +\n> +\t\tfor (i = 0;i < extra_keep->nr; i++)\n\nStyle: SP after ';' before 'i'.\n\n> +\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n> +\t\t\t\tbreak;\n> +\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n> +\t\t\tcontinue;\n> +\n> \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n> \t\t\tcontinue;\n\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index 38247afbec..553d907d34 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -196,5 +196,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n> \tgit cat-file -t $H1\n> '\n> \n> +test_expect_success 'repack --keep-pack' '\n> +\ttest_create_repo keep-pack &&\n> +\t(\n> +\t\tcd keep-pack &&\n> +\t\tfor cmit in one two three four; do\n> +\t\t\ttest_commit $cmit &&\n> +\t\t\tgit repack -d\n> +\t\tdone &&\n\nStyle: replace \"; \" before do with LF followed by a few HT.\n\nThis 'for' loop would not exit and report error if an early\ntest_commit or \"git repack -d\" fails, would it?\n\n> +\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n> +\t\ttest_line_count = 4 pack-list &&\n> +\t\tKEEP1=`head -n1 pack-list` &&\n> +\t\tKEEP4=`tail -n1 pack-list` &&\n\nStyle: $()\n\n> +\t\tgit repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n> +\t\tls .git/objects/pack/*.pack >new-counts &&\n> +\t\ttest_line_count = 3 new-counts &&\n> +\t\tgit fsck\n\nOne important invariant for this new feature is that $KEEP1 and\n$KEEP4 will both appear in new-counts file, no? Rename new-counts\nto new-pack-list and inspect the contents, not just line count,\nperhaps?\n\n> +\t)\n> +'\n> +\n> test_done\n"},{"id":"341140","messageId":"xmqqfu5d3tbn.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180306104158.6541-4-pclouds@gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-06T19:19:40Z","receivedAt":"2018-03-06T19:19:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> On linux-2.6.git, valgrind massif reports 1.6GB heap in \"pack all\"\n> case, and 535MB [1] in \"pack all except the base pack\" case. We save\n> roughly 1GB memory by excluding the base pack.\n\n;-)\n\n> gc --auto decides to do this based on an estimation of pack-objects\n> memory usage, which is quite accurate at least for the heap part, and\n> whether that fits in half of system memory (the assumption here is for\n> desktop environment where there are many other applications running).\n\nI was still confused by \"decides to do this...\" after reading the\nabove three times. If this is describing the state _with_ this\npatch applied, then \"Teach 'gc --auto' to do this automatically...\"\nwould make it clear that is what is going on.\n\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 571b5a7e3c..35ad420d5c 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -59,6 +59,11 @@ then existing packs (except those marked with a `.keep` file)\n> are consolidated into a single pack by using the `-A` option of\n> 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n> automatic consolidation of packs.\n> ++\n> +If the physical amount of memory is considered not enough for `git\n> +repack` to run smoothly, `--keep-base-pack` is enabled. This could be\n> +overridden by setting `gc.bigBasePackThreshold` which only enables\n> +`--keep-base-pack` when the base pack is larger the specified limit.\n\nI somehow find the flow of logic in these two sentences harder to\nfollow than necessary. Perhaps swapping the order may make it\neasier to grok? That is:\n\n - When gc.bigBasePackThreshold is set, packs larger than that will\n automatically be kept (i.e. not considered for repacking);\n\n - When it is not set, we try to guess how memory constrained we are,\n and behave as if the threshold were set to the size of the\n largest packfile we have (i.e. that single pack is kept).\n\nI think another and bigger reason why I found the original hard to\nread is because it is described for those who already understand\nwhat \"--keep-base-pack\" option does. Rewriting it not to require\nthe pre-existing knowledge of that option would make it a lot easier\nto grok, I would think (you may not realize it because you wrote the\nfeature and are very familiar with it, though).\n\n> +--keep-base-pack::\n> +\tAll packs except the base pack are consolidated into a single\n> +\tpack. The largest pack is considered the base pack.\n\nThis makes it sound as if packs with .keep are also repacked unless\nthey meet the threshold for \"base pack\". Is that what you actually\nimplemented?\n\nIn order to do so, [2/5] needs to allow the \"--keep-pack\" option\noverride the on-disk .keep files. In an earlier message, I wondered\nif such an arrangement is useful in some use cases; I think it is,\nand because those who do want to see the on-disk .keep files honored\ncan collect and include them in the set of packs to be kept via\n\"--keep-pack\" (after all this is an option for low-level scripting\nanyway).\n\n> +Set environment variable `GIT_TRACE` in order to see the memory usage\n> +estimation in `git gc --auto` that determines whether the base pack is\n> +kept.\n\nThis is somewhat a puzzling use of trace. It sounds more like a way\nto find out \"how\" memory usage estimation is done and arriving at a\nwrong value for those who want to debug the feature.\n\n> +static unsigned long big_base_pack_threshold;\n> +static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n\nA new symbol, which is a good addition.\n\n> +static struct packed_git *find_the_base_pack(void)\n> +{\n> +\tstruct packed_git *p, *base = NULL;\n> +\n> +\tprepare_packed_git();\n> +\n> +\tfor (p = packed_git; p; p = p->next) {\n> +\t\tif (p->pack_local &&\n> +\t\t (!base || base->pack_size < p->pack_size))\n> +\t\t\tbase = p;\n> +\t}\n> +\n> +\treturn base;\n> +}\n\nThis is finding the largest pack.\n\n> @@ -187,7 +211,101 @@ static int too_many_packs(void)\n> \treturn gc_auto_pack_limit < cnt;\n> }\n> \n> -static void add_repack_all_option(void)\n> +static inline unsigned long total_ram(void)\n\n\"inline\"? We'd rather have compiler figure it out, no?\n\n> +{\n> +\tunsigned long default_ram = 4;\n\n4 what? 4 bytes? Name it perhaps \"default_ram_gb\" or something?\n\n> +#ifdef HAVE_SYSINFO\n> +\tstruct sysinfo si;\n> +\n> +\tif (!sysinfo(&si))\n> +\t\treturn si.totalram;\n> +#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n> +\tint64_t physical_memory;\n> +\tint mib[2];\n> +\tsize_t length;\n> +\n> +\tmib[0] = CTL_HW;\n> +\tmib[1] = HW_MEMSIZE;\n> +\tlength = sizeof(int64_t);\n> +\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n> +\t\treturn physical_memory;\n> +#elif defined(GIT_WINDOWS_NATIVE)\n> +\tMEMORYSTATUSEX memInfo;\n> +\n> +\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n> +\tif (GlobalMemoryStatusEx(&memInfo))\n> +\t\treturn memInfo;ullTotalPhys;\n\nIs this legal C in Microsoft land?\n\n> +#else\n> +\tfprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n> +\t\tdefault_ram);\n> +#endif\n> +\treturn default_ram * 1024 * 1024 * 1024;\n> +}\n\nI wonder if the above should go somewhere under compat/ without\nifdef but split into separate files for individual archs (I do not\nknow the answer to this question).\n\n> +static void add_repack_all_option(struct packed_git *keep_pack)\n> {\n> \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n> \t\targv_array_push(&repack, \"-a\");\n> @@ -196,6 +314,10 @@ static void add_repack_all_option(void)\n> \t\tif (prune_expire)\n> \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n> \t}\n> +\n> +\tif (keep_pack)\n> +\t\targv_array_pushf(&repack, \"--keep-pack=%s\",\n> +\t\t\t\t basename(keep_pack->pack_name));\n> }\n> \n> static void add_repack_incremental_option(void)\n> @@ -218,9 +340,14 @@ static int need_to_gc(void)\n> \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n> \t * there is no need.\n> \t */\n> -\tif (too_many_packs())\n> -\t\tadd_repack_all_option();\n> -\telse if (too_many_loose_objects())\n> +\tif (too_many_packs()) {\n> +\t\tstruct packed_git *exclude = find_the_base_pack();\n> +\n> +\t\tif (!pack_objects_uses_too_much_memory(exclude))\n> +\t\t\texclude = NULL;\n> +\n> +\t\tadd_repack_all_option(exclude);\n\nOK, so we possibly exclude the largest one or nothing (i.e. at most\none --keep-pack is given) here.\n\n> @@ -353,6 +480,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \tconst char *name;\n> \tpid_t pid;\n> \tint daemonized = 0;\n> +\tint keep_base_pack = -1;\n> \n> \tstruct option builtin_gc_options[] = {\n> \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n> @@ -362,6 +490,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n> \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n> \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n> +\t\tOPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n> +\t\t\t N_(\"repack all other packs except the base pack\")),\n> \t\tOPT_END()\n> \t};\n> \n> @@ -427,8 +557,19 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \t\t\t */\n> \t\t\tdaemonized = !daemonize();\n> \t\t}\n> -\t} else\n> -\t\tadd_repack_all_option();\n> +\t} else {\n> +\t\tstruct packed_git *base_pack = find_the_base_pack();\n> +\t\tstruct packed_git *exclude = NULL;\n> +\n> +\t\tif (keep_base_pack != -1) {\n> +\t\t\tif (keep_base_pack)\n> +\t\t\t\texclude = base_pack;\n\nOK, --keep-base-pack option always wins if given...\n\n> +\t\t} else if (base_pack && big_base_pack_threshold &&\n> +\t\t\t base_pack->pack_size >= big_base_pack_threshold)\n> +\t\t\texclude = base_pack;\n\n...and then if the largest one is larger than the threshold, it (and\nit alone) is kept, but otherwise nothing is kept automatically.\n\nBut to those who say \"packs larger than this value is too big\" via\nconfiguration, keeping only the largest of these above-threshold\npacks would look counter-intuitive, wouldn't it, I wonder?\n\n> +\t\tadd_repack_all_option(exclude);\n> +\t}\n\n> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n> index 41b0be575d..863fdbb0fd 100755\n> --- a/t/t6500-gc.sh\n> +++ b/t/t6500-gc.sh\n> @@ -5,6 +5,13 @@ test_description='basic git gc tests\n> \n> . ./test-lib.sh\n> \n> +test_expect_success 'setup' '\n> +\t# do not let the amount of physical memory affects gc\n> +\t# behavior, make sure the pack_objects_uses_too_much_memory()\n> +\t# always returns false\n> +\tgit config gc.bigBasePackThreshold 2g\n\nHmph, that is because the configuration wins and we know the trash\nrepository will never have a pack that large. OK.\n\nI won't comment on the style issue in the remainder, as it shares\nthe same as another patch.\n\n> @@ -116,6 +123,28 @@ test_expect_success 'background auto gc respects lock for all operations' '\n> \ttest_path_is_file .git/refs/heads/should-be-loose\n> '\n> \n> +test_expect_success 'gc --keep-base-pack' '\n> +\ttest_create_repo keep-pack &&\n> +\t(\n> +\t\tcd keep-pack &&\n> +\t\tfor i in 10; do\n> +\t\t\ttest_commit $i\n> +\t\tdone &&\n> +\t\tgit gc &&\n\nThis, because of the set-up step, is guaranteed to pack everything\ninto one.\n\n> +\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n> +\t\ttest_line_count = 1 pack-list &&\n> +\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n\nAnd that is what these verify.\n\n> +\t\tfor i in 10; do\n> +\t\t\ttest_commit more-$i\n> +\t\tdone &&\n> +\t\tgit gc --keep-base-pack &&\n\nAnd we allow --keep-base-pack option to pick the largest pack (there\nis only one pack anyway) and keep it, which ...\n\n> +\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n> +\t\ttest_line_count = 2 pack-list &&\n\n... results in two packs in total. And\n\n> +\t\ttest_path_is_file $BASE_PACK &&\n\n... we make sure the first pack is left intact (unlike the earlier\n'new-count' test). This sounds sensible.\n\n> +\t\tgit fsck\n> +\t)\n> +'\n> +\n> # DO NOT leave a detached auto gc process running near the end of the\n> # test script: it can run long enough in the background to racily\n> # interfere with the cleanup in 'test_done'.\n"},{"id":"341194","messageId":"CACsJy8BW_EtxQvgL=YrCXCQY7cEWCQxgfkeH=Gd=X=uVYhPJcw@mail.gmail.com","threadId":"47943","inReplyTo":"xmqq1sgx5adl.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 2/5] repack: add --keep-pack option","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-07T10:19:50Z","receivedAt":"2018-03-07T10:20:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 7, 2018 at 1:25 AM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> +--keep-pack=::\n>> + Ignore the given pack. This is the equivalent of having\n>> + `.keep` file on the pack. Implies `--honor-pack-keep`.\n>> +\n>\n> A few questions I am not sure how I would answer:\n>\n> - Do we want to have this listed in the SYNOPSIS section, too?\n>\n> - We would want to make the SP in \"\" consistent with\n> the dash in \"\" in the same document; which way do\n> we make it uniform?\n\nProbably the latter.\n\n>\n> - Is this description clear enough to convey that we allow more\n> than one instance of this option specified, and the pack names\n> accumulate?\n\nIf a question is raised, it's probably not clear.\n\n> - Are there use cases where we want to _ignore_ on-disk \".keep\" and\n> only honor the ones given via the \"--keep-pack\" options?\n\nI can't think of one. These .keep files are originally lock files and\nignoring them sounds like a bad idea. Perhaps we could add\n--no-keep-pack later to explicit not keep a pack, ignoring .keep file\nif present?\n\n>> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n>> index 38247afbec..553d907d34 100755\n>> --- a/t/t7700-repack.sh\n>> +++ b/t/t7700-repack.sh\n>> @@ -196,5 +196,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n>> git cat-file -t $H1\n>> '\n>>\n>> +test_expect_success 'repack --keep-pack' '\n>> + test_create_repo keep-pack &&\n>> + (\n>> + cd keep-pack &&\n>> + for cmit in one two three four; do\n>> + test_commit $cmit &&\n>> + git repack -d\n>> + done &&\n>\n> Style: replace \"; \" before do with LF followed by a few HT.\n>\n> This 'for' loop would not exit and report error if an early\n> test_commit or \"git repack -d\" fails, would it?\n\nYeah. I guess I'll just unroll the loop.\n\n>> + git repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n>> + ls .git/objects/pack/*.pack >new-counts &&\n>> + test_line_count = 3 new-counts &&\n>> + git fsck\n>\n> One important invariant for this new feature is that $KEEP1 and\n> $KEEP4 will both appear in new-counts file, no? Rename new-counts\n> to new-pack-list and inspect the contents, not just line count,\n> perhaps?\n\nOK\n-- \nDuy\n"},{"id":"341195","messageId":"nycvar.QRO.7.76.6.1803071140130.20700@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47943","inReplyTo":"20180306104158.6541-4-pclouds@gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-07T10:48:45Z","receivedAt":"2018-03-07T10:49:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Tue, 6 Mar 2018, Nguyễn Thái Ngọc Duy wrote:\n\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 77fa720bd0..273657ddf4 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -187,7 +211,101 @@ static int too_many_packs(void)\n> \treturn gc_auto_pack_limit < cnt;\n> }\n> \n> -static void add_repack_all_option(void)\n> +static inline unsigned long total_ram(void)\n> +{\n> +\tunsigned long default_ram = 4;\n> +#ifdef HAVE_SYSINFO\n> +\tstruct sysinfo si;\n> +\n> +\tif (!sysinfo(&si))\n> +\t\treturn si.totalram;\n> +#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n> +\tint64_t physical_memory;\n> +\tint mib[2];\n> +\tsize_t length;\n> +\n> +\tmib[0] = CTL_HW;\n> +\tmib[1] = HW_MEMSIZE;\n> +\tlength = sizeof(int64_t);\n> +\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n> +\t\treturn physical_memory;\n> +#elif defined(GIT_WINDOWS_NATIVE)\n> +\tMEMORYSTATUSEX memInfo;\n> +\n> +\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n> +\tif (GlobalMemoryStatusEx(&memInfo))\n> +\t\treturn memInfo;ullTotalPhys;\n\nThis fails to compile:\n\nbuiltin/gc.c: In function 'total_ram':\nbuiltin/gc.c:235:10: error: incompatible types when returning type 'MEMORYSTATUSEX {aka struct _MEMORYSTATUSEX}' but 'long unsigned int' was expected\n\t return memInfo;ullTotalPhys;\n\t ^~~~~~~\nbuiltin/gc.c:234:2: error: this 'if' clause does not guard... [-Werror=misleading-indentation]\n\t if (GlobalMemoryStatusEx(&memInfo))\n\t ^~\nbuiltin/gc.c:235:18: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'\n\t return memInfo;ullTotalPhys;\n\t ^~~~~~~~~~~~\nbuiltin/gc.c:235:18: error: 'ullTotalPhys' undeclared (first use in this function)\nbuiltin/gc.c:235:18: note: each undeclared identifier is reported only once for each function it appears in\n\nI suspect that the first semicolon wanted to be a period instead. At least\nit fixes the build here (that's all I can test, I'm at GitMerge and miss a\nvery interesting discussion about the serialized commit graph to write\nthis):\n\n-- snip --\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 4f46a99ab54..9c12f1ee9af 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -232,7 +232,7 @@ static inline unsigned long total_ram(void)\n \n \tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n \tif (GlobalMemoryStatusEx(&memInfo))\n-\t\treturn memInfo;ullTotalPhys;\n+\t\treturn memInfo.ullTotalPhys;\n #else\n \tfprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n \t\tdefault_ram);\n\n-- snap --\n\nJunio, may I ask you to put this into a SQUASH??? commit so that the\nWindows build no longer fails?\n\nThanks,\nDscho"},{"id":"341196","messageId":"CACsJy8C2j2dLozhHf4zE--KYPZ2VjOW2tobmczh3rf6L==Q1vw@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqfu5d3tbn.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-07T10:48:59Z","receivedAt":"2018-03-07T10:49:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 7, 2018 at 2:19 AM, Junio C Hamano wrote:\n>> +--keep-base-pack::\n>> + All packs except the base pack are consolidated into a single\n>> + pack. The largest pack is considered the base pack.\n>\n> This makes it sound as if packs with .keep are also repacked unless\n> they meet the threshold for \"base pack\". Is that what you actually\n> implemented?\n\nCopy/paste problem. That is, I copied this from --auto description,\nbut I missed the \"(except those marked with a `.keep` file)\" part. No,\nI don't think ignoring .keep files is a good idea, at least no by\ndefault.\n\n> In order to do so, [2/5] needs to allow the \"--keep-pack\" option\n> override the on-disk .keep files. In an earlier message, I wondered\n> if such an arrangement is useful in some use cases; I think it is,\n> and because those who do want to see the on-disk .keep files honored\n> can collect and include them in the set of packs to be kept via\n> \"--keep-pack\" (after all this is an option for low-level scripting\n> anyway).\n\nAt gc level I don't think we need to allow this. But yeah git-repack\nhas --pack-kept-objects to ignore .keep. If they specify this, then\nrepack should ignore .keep (but still follow whatever --keep-pack is\nspecified). There's some interesting interaction between .keep and\npack bitmap feature in pack-objects though. I'm not so sure what\nhappens down there yet.\n\n>> +Set environment variable `GIT_TRACE` in order to see the memory usage\n>> +estimation in `git gc --auto` that determines whether the base pack is\n>> +kept.\n>\n> This is somewhat a puzzling use of trace. It sounds more like a way\n> to find out \"how\" memory usage estimation is done and arriving at a\n> wrong value for those who want to debug the feature.\n\nYeah. I'm not sure if this estimation could be really problematic that\npeople need to debug this way. A cleaner way (if we think people will\nneed this often) is just add a new option in \"git gc\" to report this\nestimation breakdown and do nothing else.\n\n>> +static struct packed_git *find_the_base_pack(void)\n>> +{\n>> + struct packed_git *p, *base = NULL;\n>> +\n>> + prepare_packed_git();\n>> +\n>> + for (p = packed_git; p; p = p->next) {\n>> + if (p->pack_local &&\n>> + (!base || base->pack_size < p->pack_size))\n>> + base = p;\n>> + }\n>> +\n>> + return base;\n>> +}\n>\n> This is finding the largest pack.\n\nThe discussion on .keep files raises one question for me, what if the\nlargest pack already has a .keep file, do we still consider it the\nbase pack, or should we find the next largest non-kept pack?\n\nI'm guessing we keep things simple here and ignore .keep files.\n\n>> +#ifdef HAVE_SYSINFO\n>> + struct sysinfo si;\n>> +\n>> + if (!sysinfo(&si))\n>> + return si.totalram;\n>> +#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n>> + int64_t physical_memory;\n>> + int mib[2];\n>> + size_t length;\n>> +\n>> + mib[0] = CTL_HW;\n>> + mib[1] = HW_MEMSIZE;\n>> + length = sizeof(int64_t);\n>> + if (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n>> + return physical_memory;\n>> +#elif defined(GIT_WINDOWS_NATIVE)\n>> + MEMORYSTATUSEX memInfo;\n>> +\n>> + memInfo.dwLength = sizeof(MEMORYSTATUSEX);\n>> + if (GlobalMemoryStatusEx(&memInfo))\n>> + return memInfo;ullTotalPhys;\n>\n> Is this legal C in Microsoft land?\n\nThat's the problem with writing code without a way to test it. At\nleast travis helped catch a compiler bug on mac.\n\nI'm torn between just #error here and let Windows/Mac guys implement\nit (which they may scream \"too much work, I don't wanna\") but if I\nhelp write something first, yes things are potentially broken and need\nverification from those guys. I guess I'll just fix this up and hope\nnon-linux guys do the rest.\n\n>> +#else\n>> + fprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n>> + default_ram);\n>> +#endif\n>> + return default_ram * 1024 * 1024 * 1024;\n>> +}\n>\n> I wonder if the above should go somewhere under compat/ without\n> ifdef but split into separate files for individual archs (I do not\n> know the answer to this question).\n\nMy first choice too. I chose this way after seeing online_cpus()\nthread-utils.c. Not sure which way is best either.\n\n>> @@ -427,8 +557,19 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n>> */\n>> daemonized = !daemonize();\n>> }\n>> - } else\n>> - add_repack_all_option();\n>> + } else {\n>> + struct packed_git *base_pack = find_the_base_pack();\n>> + struct packed_git *exclude = NULL;\n>> +\n>> + if (keep_base_pack != -1) {\n>> + if (keep_base_pack)\n>> + exclude = base_pack;\n>\n> OK, --keep-base-pack option always wins if given...\n>\n>> + } else if (base_pack && big_base_pack_threshold &&\n>> + base_pack->pack_size >= big_base_pack_threshold)\n>> + exclude = base_pack;\n>\n> ...and then if the largest one is larger than the threshold, it (and\n> it alone) is kept, but otherwise nothing is kept automatically.\n>\n> But to those who say \"packs larger than this value is too big\" via\n> configuration, keeping only the largest of these above-threshold\n> packs would look counter-intuitive, wouldn't it, I wonder?\n\nI think I'll just clarify this in the document. There may be a use\ncase for keeping multiple large packs, but I don't see it (*). We can\ndeal with it when it comes.\n\n(*) Well I see one. In submodule setting, we if merge object stores of\nall submodules back to the supermodule, we have multiple base packs\nand probably want to keep them that way. It's still a long way to get\nthere (not even sure if submodule people want to get there)\n\n>> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n>> index 41b0be575d..863fdbb0fd 100755\n>> --- a/t/t6500-gc.sh\n>> +++ b/t/t6500-gc.sh\n>> @@ -5,6 +5,13 @@ test_description='basic git gc tests\n>>\n>> . ./test-lib.sh\n>>\n>> +test_expect_success 'setup' '\n>> + # do not let the amount of physical memory affects gc\n>> + # behavior, make sure the pack_objects_uses_too_much_memory()\n>> + # always returns false\n>> + git config gc.bigBasePackThreshold 2g\n>\n> Hmph, that is because the configuration wins and we know the trash\n> repository will never have a pack that large. OK.\n\nCredit goes to travis linux32 job. I wouldn't notice this otherwise.\n\nA thought came across me when I wrote this though. Should we support\nspecial value \"infinite\" (or just \"max\")- in our config code?\n\nThe use of super large gc.bigBasePackThreshold to disable this keeping\nbase pack is intended. But I can't go too high here it may break\nlimits on 32 bit platforms. And 2g sounds really arbitrary.\n-- \nDuy\n"},{"id":"341223","messageId":"xmqqd10f20k6.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"CACsJy8C2j2dLozhHf4zE--KYPZ2VjOW2tobmczh3rf6L==Q1vw@mail.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-07T18:38:33Z","receivedAt":"2018-03-07T18:38:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n>>> +Set environment variable `GIT_TRACE` in order to see the memory usage\n>>> +estimation in `git gc --auto` that determines whether the base pack is\n>>> +kept.\n>>\n>> This is somewhat a puzzling use of trace. It sounds more like a way\n>> to find out \"how\" memory usage estimation is done and arriving at a\n>> wrong value for those who want to debug the feature.\n>\n> Yeah. I'm not sure if this estimation could be really problematic that\n> people need to debug this way. A cleaner way (if we think people will\n> need this often) is just add a new option in \"git gc\" to report this\n> estimation breakdown and do nothing else.\n\nActually after reading the implementation and seeing what it does, I\npersonally do not have any problem with the way GIT_TRACE is used\nfor this purpose in this patch. I am not sure how interesting the\ninformation given by that codepath in real life; it looks even less\nintereseting than say what comes out of \"verify-pack --stat\".\n\n>> This is finding the largest pack.\n>\n> The discussion on .keep files raises one question for me, what if the\n> largest pack already has a .keep file, do we still consider it the\n> base pack, or should we find the next largest non-kept pack?\n>\n> I'm guessing we keep things simple here and ignore .keep files.\n\nI agree that it is a sensible design decision.\n\n>>> +#elif defined(GIT_WINDOWS_NATIVE)\n>>> + MEMORYSTATUSEX memInfo;\n>>> +\n>>> + memInfo.dwLength = sizeof(MEMORYSTATUSEX);\n>>> + if (GlobalMemoryStatusEx(&memInfo))\n>>> + return memInfo;ullTotalPhys;\n>>\n>> Is this legal C in Microsoft land?\n>\n> That's the problem with writing code without a way to test it. At\n> least travis helped catch a compiler bug on mac.\n>\n> I'm torn between just #error here and let Windows/Mac guys implement\n> it (which they may scream \"too much work, I don't wanna\") but if I\n> help write something first, yes things are potentially broken and need\n> verification from those guys. I guess I'll just fix this up and hope\n> non-linux guys do the rest.\n\nYup, we all collaborate and help in ways each of us can. None of us\ncan be expected to do any more than that ;-)\n\n>> I wonder if the above should go somewhere under compat/ without\n>> ifdef but split into separate files for individual archs (I do not\n>> know the answer to this question).\n>\n> My first choice too. I chose this way after seeing online_cpus()\n> thread-utils.c. Not sure which way is best either.\n\nOK.\n\n>> But to those who say \"packs larger than this value is too big\" via\n>> configuration, keeping only the largest of these above-threshold\n>> packs would look counter-intuitive, wouldn't it, I wonder?\n>\n> I think I'll just clarify this in the document. There may be a use\n> case for keeping multiple large packs, but I don't see it (*). We can\n> deal with it when it comes.\n\nWhen the project's history grows too much, a large pack that holds\nits first 10 years of stuff, together with another one that holds\nits second 20 years of stuff, may both be larger than the threshold\nand want to be kept. If we pick only the largest one, we would\nexplode the other one and repack together with loose objects.\n\nBut realistically, those who would want to control the way in which\ntheir repository is packed to such a degree are very likely to add\n\".keep\" files to these two packfiles themselves, so the above would\nprobably not a concern. Perhaps we shouldn't do the \"automatically\npick the largest one and exclude from repacking\" when there is a\npackfile that is marked with \".keep\"?\n\n> The use of super large gc.bigBasePackThreshold to disable this keeping\n> base pack is intended. But I can't go too high here it may break\n> limits on 32 bit platforms. And 2g sounds really arbitrary.\n\nYou could use 42m instead to clarify that it really is an arbitrary\nthreshold that was chosen only for the purpose of this test perhaps?\n;-)\n"},{"id":"341224","messageId":"xmqq8tb320h3.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"nycvar.QRO.7.76.6.1803071140130.20700@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-07T18:40:24Z","receivedAt":"2018-03-07T18:40:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Junio, may I ask you to put this into a SQUASH??? commit so that the\n> Windows build no longer fails?\n\nThanks for a reminder; I also spotted it (I first thought I screwed\nup in my editor while reviewing and then went back to the original\non the list) and sent out a response, but then by that time I was\nalready far into the day's integration cycle.\n\nWill queue a SQUASH??? at the tip.\n"},{"id":"341269","messageId":"20180308114232.10508-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180303024706.31465-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 00/12] Reduce pack-objects memory footprint","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:20Z","receivedAt":"2018-03-08T11:42:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v3 cleans up a bit to avoid the horrible macros that v2 adds, and\nadds some more documentation for this struct since it's getting harder\nto just look at the struct and see what field is related to what.\n\nv3 also adds three more patches and reduces another 16 bytes (total\nstruct reduction now is 41%). After this there's hardly anything else I\ncould do. Two 64-bit fields left, but even if I shrink them, I'd lose\nit to padding. There's still one possibility to share in_pack_offset\nwith idx.offset, but it's risky.\n\nThese three patches are made to optimize for the common case. The\nincommon cases will suffer some performance loss:\n\n- 10/12 limits the cached compressed delta size to 64k (default 1000\n bytes). If you normally have lots of huge deltas, you're going to\n take a hit because more deltas must be recreated at writing phase.\n Note that it does not stop pack-objects from creating deltas larger\n than 64k.\n\n- 11/12 reduces uncompressed object size to 4GB. Whenever we need to\n read object size of those larger than that, we read the pack again\n to retrieve the information, which is much slower than accessing a\n piece of memory. Again I'm assuming these giant blobs are really\n really rare that this performance hit won't matter.\n\n- 12/12 is similar to 11/12 and reduces uncompressed delta size to\n 4GB. Frankly a 4GB delta is still ridiculous, but I don't think we\n gain more by shrinking it further. If your packs have one of those\n giant deltas, it still works, delta_size will be read back from the\n pack again.\n\nThe following interdiff does _NOT_ cover the new patches, just the\nfirst nine that v2 has.\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 55f19a1f18..82a4a95888 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,19 +29,13 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n-#define DELTA(obj) \\\n-\t((obj)->delta_idx ? &to_pack.objects[(obj)->delta_idx - 1] : NULL)\n-#define DELTA_CHILD(obj) \\\n-\t((obj)->delta_child_idx ? &to_pack.objects[(obj)->delta_child_idx - 1] : NULL)\n-#define DELTA_SIBLING(obj) \\\n-\t((obj)->delta_sibling_idx ? &to_pack.objects[(obj)->delta_sibling_idx - 1] : NULL)\n-\n-#define CLEAR_DELTA(obj) (obj)->delta_idx = 0\n-#define CLEAR_DELTA_CHILD(obj) (obj)->delta_child_idx = 0\n-#define CLEAR_DELTA_SIBLING(obj) (obj)->delta_sibling_idx = 0\n-\n-#define SET_DELTA(obj, val) (obj)->delta_idx = ((val) - to_pack.objects) + 1\n-#define SET_DELTA_CHILD(obj, val) (obj)->delta_child_idx = ((val) - to_pack.objects) + 1\n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -381,7 +375,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -492,7 +486,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!IN_PACK(&to_pack, entry))\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n@@ -557,7 +551,7 @@ static enum write_one_status write_one(struct hashfile *f,\n \t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\tCLEAR_DELTA(e);\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -672,8 +666,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tCLEAR_DELTA_CHILD(&objects[i]);\n-\t\tCLEAR_DELTA_SIBLING(&objects[i]);\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -1067,19 +1061,8 @@ static int want_object_in_pack(const struct object_id *oid,\n \n \twant = 1;\n done:\n-\tif (want && *found_pack && !(*found_pack)->index) {\n-\t\tstruct packed_git *p = *found_pack;\n-\n-\t\tif (to_pack.in_pack_count >= (1 << OE_IN_PACK_BITS))\n-\t\t\tdie(_(\"too many packs to handle in one go. \"\n-\t\t\t \"Please add .keep files to exclude\\n\"\n-\t\t\t \"some pack files and keep the number \"\n-\t\t\t \"of non-kept files below %d.\"),\n-\t\t\t 1 << OE_IN_PACK_BITS);\n-\n-\t\tp->index = to_pack.in_pack_count++;\n-\t\tto_pack.in_pack[p->index] = p;\n-\t}\n+\tif (want && *found_pack && !(*found_pack)->index)\n+\t\toe_add_pack(&to_pack, *found_pack);\n \n \treturn want;\n }\n@@ -1104,9 +1087,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tif (found_pack->index <= 0)\n-\t\t\tdie(\"BUG: found_pack should be NULL instead of having non-positive index\");\n-\t\tentry->in_pack_idx = found_pack->index;\n+\t\toe_set_in_pack(entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1431,8 +1412,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (IN_PACK(&to_pack, entry)) {\n-\t\tstruct packed_git *p = IN_PACK(&to_pack, entry);\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1567,8 +1548,8 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n-\tconst struct packed_git *a_in_pack = IN_PACK(&to_pack, a);\n-\tconst struct packed_git *b_in_pack = IN_PACK(&to_pack, b);\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n \tif (!a_in_pack && !b_in_pack)\n@@ -1609,12 +1590,12 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\telse\n \t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tCLEAR_DELTA(entry);\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(IN_PACK(&to_pack, entry), entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1884,8 +1865,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && IN_PACK(&to_pack, trg_entry) &&\n-\t IN_PACK(&to_pack, trg_entry) == IN_PACK(&to_pack, src_entry) &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -2994,16 +2975,6 @@ static int option_parse_unpack_unreachable(const struct option *opt,\n \treturn 0;\n }\n \n-static void init_in_pack_mapping(struct packing_data *to_pack)\n-{\n-\t/* let IN_PACK() return NULL if in_pack_idx is zero */\n-\tto_pack->in_pack[to_pack->in_pack_count++] = NULL;\n-\t/*\n-\t * the rest is lazily initialized only for packs that we want\n-\t * in want_object_in_pack().\n-\t */\n-}\n-\n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n@@ -3236,7 +3207,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t}\n \t}\n-\tinit_in_pack_mapping(&to_pack);\n+\n+\t/* make sure IN_PACK(0) return NULL */\n+\toe_add_pack(&to_pack, NULL);\n \n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 1360a93311..256a63f892 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,7 +64,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tIN_PACK_POS(to_pack, entry) = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (entry->type) {\n \t\tcase OBJ_COMMIT:\n@@ -149,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn IN_PACK_POS(writer.to_pack, entry);\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f21479fe16..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = IN_PACK_POS(mapping, oe) + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-objects.h b/pack-objects.h\nindex a57aca5f03..3c15cf7b23 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,15 +1,9 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n-#define OE_DFS_STATE_BITS 2\n-#define OE_DEPTH_BITS 12\n-#define OE_IN_PACK_BITS 14\n-\n-#define IN_PACK_POS(to_pack, obj) \\\n-\t(to_pack)->in_pack_pos[(struct object_entry *)(obj) - (to_pack)->objects]\n-\n-#define IN_PACK(to_pack, obj) \\\n-\t(to_pack)->in_pack[(obj)->in_pack_idx]\n+#define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t14\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -28,6 +22,51 @@ enum dfs_state {\n * The size of struct nearly determines pack-objects's memory\n * consumption. This struct is packed tight for that reason. When you\n * add or reorder something in this struct, think a bit about this.\n+ *\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 and\n+ * is only valid after the object is written down and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size is not\n+ * cached (ie. raw data in a pack) but available via revindex.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n+ * the location of the object in the source pack, with or without\n+ * header.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, \"delta\" points to\n+ * the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached. Delta recreation technically only depends on \"delta\"\n+ * pointer, but delta_size is still used to verify it's the same as\n+ * before.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -103,4 +142,109 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n+static inline unsigned int oe_add_pack(struct packing_data *pack,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t \"Please add .keep files to exclude\\n\"\n+\t\t \"some pack files and keep the number \"\n+\t\t \"of non-kept files below %d.\"),\n+\t\t 1 << OE_IN_PACK_BITS);\n+\tif (p) {\n+\t\tif (p->index > 0)\n+\t\t\tdie(\"BUG: this packed is already indexed\");\n+\t\tp->index = pack->in_pack_count;\n+\t}\n+\tpack->in_pack[pack->in_pack_count] = p;\n+\treturn pack->in_pack_count++;\n+}\n+\n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack[e->in_pack_idx];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (p->index <= 0)\n+\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t \"instead of having non-positive index\");\n+\te->in_pack_idx = p->index;\n+\n+}\n+\n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n\nNguyễn Thái Ngọc Duy (12):\n pack-objects: a bit of document about struct object_entry\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: note about in_pack_header_size\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: reorder 'hash' to pack struct object_entry\n pack-objects: shrink z_delta_size field in struct object_entry\n pack-objects: shrink size field in struct object_entry\n pack-objects: shrink delta_size field in struct object_entry\n\n Documentation/config.txt | 4 +-\n Documentation/git-pack-objects.txt | 13 +-\n Documentation/git-repack.txt | 4 +-\n builtin/pack-objects.c | 269 +++++++++++++++++----------\n cache.h | 3 +\n object.h | 1 -\n pack-bitmap-write.c | 8 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.h | 288 ++++++++++++++++++++++++++---\n 10 files changed, 460 insertions(+), 136 deletions(-)\n\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341270","messageId":"20180308114232.10508-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 01/12] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:21Z","receivedAt":"2018-03-08T11:42:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what. This also documents the holes in this struct\naccording to pahole.\n\nA couple of notes on shrinking the struct:\n\n1) The reader may notice one thing from this document and the shrinking\nbusiness. If \"delta\" is NULL, all other delta-related fields should be\nirrelevant. We could group all these in a separate struct and replace\nthem all with a pointer to this struct (allocated separately).\n\nThis does not help much though since 85% of objects are deltified\n(source: linux-2.6.git). The gain is only from non-delta objects, which\nis not that significant.\n\n2) The field in_pack_offset and idx.offset could be merged. But we need\nto be very careful. Up until the very last phase (object writing),\nidx.offset is not used and can hold in_pack_offset. Then idx.offset will\nbe updated with _destination pack's_ offset, not source's. But since we\nalways write delta's bases first, and we only use in_pack_offset in\nwriting phase when we reuse objects, we should be ok?\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..f834ead541 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,52 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 and\n+ * is only valid after the object is written down and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size is not\n+ * cached (ie. raw data in a pack) but available via revindex.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n+ * the location of the object in the source pack, with or without\n+ * header.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, \"delta\" points to\n+ * the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached. Delta recreation technically only depends on \"delta\"\n+ * pointer, but delta_size is still used to verify it's the same as\n+ * before.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n@@ -28,6 +74,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n+\t/* XXX 28 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -40,6 +87,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n+\t/* size: 136, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341271","messageId":"20180308114232.10508-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 02/12] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:22Z","receivedAt":"2018-03-08T11:43:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This saves 8 bytes in sizeof(struct object_entry). On a large\nrepository like linux-2.6.git (6.5M objects), this saves us 52MB\nmemory.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 14 ++++++++++++--\n cache.h | 2 ++\n object.h | 1 -\n pack-objects.h | 8 ++++----\n 4 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..fd217cb51f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n@@ -1580,6 +1586,10 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t */\n \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n \t\t\t\t\t &entry->size);\n+\t} else {\n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->type = type;\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-objects.h b/pack-objects.h\nindex f834ead541..85b01b66da 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -60,11 +60,11 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n+\tunsigned type:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n@@ -74,7 +74,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \n-\t/* XXX 28 bits hole, try to pack */\n+\t/* XXX 22 bits hole, try to pack */\n \t/*\n \t * State flags for depth-first search used for analyzing delta cycles.\n \t *\n@@ -87,7 +87,7 @@ struct object_entry {\n \t\tDFS_DONE\n \t} dfs_state;\n \tint depth;\n-\t/* size: 136, padding: 4 */\n+\t/* size: 128, padding: 4 */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341272","messageId":"20180308114232.10508-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 03/12] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:23Z","receivedAt":"2018-03-08T11:43:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 33 ++++++++++++++++++++-------------\n 2 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fd217cb51f..a4dbb40824 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 85b01b66da..628c45871c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -73,21 +88,13 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 20 bits hole, try to pack */\n \n-\t/* XXX 22 bits hole, try to pack */\n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n-\t/* size: 128, padding: 4 */\n+\n+\t/* size: 120 */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341273","messageId":"20180308114232.10508-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 04/12] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:24Z","receivedAt":"2018-03-08T11:43:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This does not give us any saving due to padding. But we will be able\nto save once we cut 4 bytes out of this struct in a subsequent patch.\n\nBecause of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 8 +++-----\n 5 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a4dbb40824..cfd97da7db 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth > (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS));\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 628c45871c..4b17402953 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -89,12 +90,9 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* XXX 20 bits hole, try to pack */\n-\n-\tint depth;\n-\n-\t/* size: 120 */\n+\t/* size: 120, bit_padding: 8 bits */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341274","messageId":"20180308114232.10508-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 05/12] pack-objects: note about in_pack_header_size","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:25Z","receivedAt":"2018-03-08T11:43:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Object header in a pack is packed really tight (see\npack-format.txt). Even with 8 bytes length, we need 9-10 bytes most,\nplus a hash (20 bytes). Which means this field only needs to store a\nnumber as big as 32 (5 bits).\n\nThis is trickier to pack tight though since a new hash algorithm is\ncoming, the number of bits needed may quickly increase. So leave it\nfor now.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 4b17402953..2ccd6359d2 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -78,7 +78,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n-\tunsigned char in_pack_header_size;\n+\tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341275","messageId":"20180308114232.10508-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 06/12] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:26Z","receivedAt":"2018-03-08T11:43:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot). This saves us 8 bytes in struct object_entry.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 18 ++++++++++++++++--\n 5 files changed, 27 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex cfd97da7db..7bb5544883 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -878,7 +878,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..256a63f892 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (entry->type) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 2ccd6359d2..9ab0ce300d 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -77,7 +77,6 @@ struct object_entry {\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -92,7 +91,7 @@ struct object_entry {\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 120, bit_padding: 8 bits */\n+\t/* size: 112, bit_padding: 8 bits */\n };\n \n struct packing_data {\n@@ -101,6 +100,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -131,4 +132,17 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341276","messageId":"20180308114232.10508-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 07/12] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:27Z","receivedAt":"2018-03-08T11:43:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index isntead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 16k. For now if you hit\n16k pack files limit, pack-objects will simply fail [1].\n\nThis technically saves 7 bytes. But we don't see any of that in\npractice due to padding. The saving becomes real when we pack this\nstruct tighter later.\n\n[1] The escape hatch is .keep file to limit the non-kept pack files\n below 16k limit. Then you can go for another pack-objects run to\n combine another 16k pack files. Repeat until you're satisfied.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++++\n builtin/pack-objects.c | 40 +++++++++++++++---------\n cache.h | 1 +\n pack-objects.h | 49 ++++++++++++++++++++++++++++--\n 4 files changed, 83 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 3503c9e3e6..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7bb5544883..7df525e201 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n@@ -1024,7 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1047,11 +1049,16 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index)\n+\t\toe_add_pack(&to_pack, *found_pack);\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1074,7 +1081,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1406,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1542,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1587,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1857,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -3191,6 +3200,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\t/* make sure IN_PACK(0) return NULL */\n+\toe_add_pack(&to_pack, NULL);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 9ab0ce300d..59c44b3420 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t14\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -18,6 +19,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 and\n@@ -66,7 +71,6 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -78,6 +82,7 @@ struct object_entry {\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n@@ -89,9 +94,12 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\n+\t/* XXX 8 bits hole, try to pack */\n+\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112, bit_padding: 8 bits */\n+\t/* size: 112, padding: 4, bit_padding: 18 bits */\n };\n \n struct packing_data {\n@@ -102,6 +110,8 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -145,4 +155,39 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline unsigned int oe_add_pack(struct packing_data *pack,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t \"Please add .keep files to exclude\\n\"\n+\t\t \"some pack files and keep the number \"\n+\t\t \"of non-kept files below %d.\"),\n+\t\t 1 << OE_IN_PACK_BITS);\n+\tif (p) {\n+\t\tif (p->index > 0)\n+\t\t\tdie(\"BUG: this packed is already indexed\");\n+\t\tp->index = pack->in_pack_count;\n+\t}\n+\tpack->in_pack[pack->in_pack_count] = p;\n+\treturn pack->in_pack_count++;\n+}\n+\n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack[e->in_pack_idx];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (p->index <= 0)\n+\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t \"instead of having non-positive index\");\n+\te->in_pack_idx = p->index;\n+\n+}\n+\n #endif\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341277","messageId":"20180308114232.10508-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 08/12] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:28Z","receivedAt":"2018-03-08T11:43:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Notice that packing_data::nr_objects is uint32_t, we could only handle\nmaximum 4G objects and can address all of them with an uint32_t. If we\nuse a pointer here, we waste 4 bytes on 64 bit architecture.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\nThere are holes in this struct but this patch is already big. Struct\npacking can be done separately. Even with holes, we save 8 bytes per\nobject_entry.\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 71 ++++++++++++++++++++++---\n 2 files changed, 127 insertions(+), 60 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7df525e201..82a4a95888 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -488,7 +494,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (entry->type != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -540,12 +546,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -607,34 +613,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -645,7 +651,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -660,8 +666,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -671,11 +677,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1498,10 +1504,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\tentry->type = entry->in_pack_type;\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1572,17 +1578,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1624,7 +1632,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1649,7 +1657,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1672,7 +1680,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1687,7 +1695,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1870,7 +1878,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1946,7 +1954,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1975,7 +1983,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1984,13 +1992,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2059,7 +2067,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2109,7 +2117,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2117,7 +2125,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2438,7 +2446,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 59c44b3420..1c0ad4c9ef 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -72,11 +72,13 @@ struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n+\t/* XXX 4 bytes hole, try to pack */\n+\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -99,7 +101,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 112, padding: 4, bit_padding: 18 bits */\n+\t/* size: 104, padding: 4, bit_padding: 18 bits */\n };\n \n struct packing_data {\n@@ -190,4 +192,61 @@ static inline void oe_set_in_pack(struct object_entry *e,\n \n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341278","messageId":"20180308114232.10508-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 09/12] pack-objects: reorder 'hash' to pack struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:29Z","receivedAt":"2018-03-08T11:43:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 1c0ad4c9ef..3c15cf7b23 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -77,12 +77,10 @@ struct object_entry {\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\t/* XXX 4 bytes hole, try to pack */\n-\n+\tuint32_t hash;\t\t\t/* name hint hash */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -101,7 +99,7 @@ struct object_entry {\n \n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 104, padding: 4, bit_padding: 18 bits */\n+\t/* size: 96, bit_padding: 18 bits */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341279","messageId":"20180308114232.10508-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 10/12] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:30Z","receivedAt":"2018-03-08T11:43:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 16 bits, so you can only cache 65kb deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nThis saves us 8 bytes (some from previous bit padding).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 22 ++++++++++++++++------\n pack-objects.h | 11 ++++++++---\n 3 files changed, 26 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 82a4a95888..39920061e9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2105,12 +2105,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tentry->z_delta_size = size;\n+\t\t\tif (entry->z_delta_size == size) {\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3089,6 +3096,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (depth > (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS));\n+\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n+\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n+\t\t 1 << OE_Z_DELTA_BITS);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 3c15cf7b23..cbb39ab568 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n+#define OE_Z_DELTA_BITS\t\t16\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -80,7 +81,6 @@ struct object_entry {\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -93,13 +93,18 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n-\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n \t/* XXX 8 bits hole, try to pack */\n \n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n+\t/*\n+\t * if delta_data contains a compressed delta, this contains\n+\t * the compressed length\n+\t*/\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \n-\t/* size: 96, bit_padding: 18 bits */\n+\t/* size: 88, bit_padding: 2 bits */\n };\n \n struct packing_data {\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341280","messageId":"20180308114232.10508-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 11/12] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:31Z","receivedAt":"2018-03-08T11:43:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressd object is larger than\n4GB (partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size is\nsmaller than this limit.\n\nShrink size field down to 32 bits [1] and one overflow bit. If the size\nis too large, we read it back from disk.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nThere's no actual saving from this due to holes. Which should be gone in\nthe next patch.\n\n[1] it's actually already 32 bits on 64-bit Windows\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 49 ++++++++++++++++++++++++++----------------\n pack-objects.h | 48 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 77 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39920061e9..db040e95db 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -274,7 +274,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (entry->type == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +384,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = oe_size(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +407,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1412,6 +1413,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1431,13 +1434,14 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n+\t\toe_set_size(entry, size);\n \n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n@@ -1505,7 +1509,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\tentry->type = entry->in_pack_type;\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = oe_size(entry);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1513,14 +1517,17 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (entry->type) {\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = get_size_from_delta(p, &w_curs,\n+\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n-\t\t\tif (entry->size == 0)\n+\t\t\toe_set_size(entry, size);\n+\t\t\tif (oe_size_less_than(entry, 1))\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1535,13 +1542,14 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\tentry->type = sha1_object_info(entry->idx.oid.hash, &size);\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n \t * as a preferred base. Doing so can result in a larger\n \t * pack file, but the transfer will still take place.\n \t */\n+\toe_set_size(entry, size);\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1589,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1602,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,11 +1612,13 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\t\t\t\t &size);\n+\t\toe_set_size(entry, size);\n \t} else {\n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->type = type;\n+\t\toe_set_size(entry, size);\n \t}\n }\n \n@@ -1748,7 +1759,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (oe_size_greater_than(entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1786,8 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tunsigned long a_size = oe_size(a);\n+\tunsigned long b_size = oe_size(b);\n \n \tif (a->type > b->type)\n \t\treturn -1;\n@@ -1788,9 +1801,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1877,7 +1890,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = oe_size(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1902,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = oe_size(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2022,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += oe_size(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2472,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (oe_size_less_than(entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.h b/pack-objects.h\nindex cbb39ab568..0253df6cd4 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -71,7 +71,11 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size_;\n+\n+\t/* XXX 4 bytes hole, try to pack */\n+\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n@@ -93,6 +97,7 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned size_valid:1;\n \n \t/* XXX 8 bits hole, try to pack */\n \n@@ -104,7 +109,7 @@ struct object_entry {\n \t*/\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \n-\t/* size: 88, bit_padding: 2 bits */\n+\t/* size: 88, bit_padding: 1 bits */\n };\n \n struct packing_data {\n@@ -252,4 +257,43 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+static inline unsigned long oe_size(const struct object_entry *e)\n+{\n+\tif (e->size_valid) {\n+\t\treturn e->size_;\n+\t} else {\n+\t\tunsigned long size;\n+\n+\t\tsha1_object_info(e->idx.oid.hash, &size);\n+\t\treturn size;\n+\t}\n+}\n+\n+static inline int oe_size_less_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ < limit;\n+\tif (limit > maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 1;\n+\treturn oe_size(e) < limit;\n+}\n+\n+static inline int oe_size_greater_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ > limit;\n+\tif (limit <= maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 1;\n+\treturn oe_size(e) > limit;\n+}\n+\n+static inline void oe_set_size(struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\te->size_ = size;\n+\te->size_valid = e->size_ == size;\n+}\n+\n #endif\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341281","messageId":"20180308114232.10508-13-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH/RFC v3 12/12] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-08T11:42:32Z","receivedAt":"2018-03-08T11:43:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n31 bits with one overflow bit.\n\nIf we encounter an existing delta larger than 2GB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 24 ++++++++++++++----------\n pack-objects.h | 30 +++++++++++++++++++++++++-----\n 2 files changed, 39 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex db040e95db..0f65e0f243 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,10 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -140,7 +142,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -291,14 +293,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1509,7 +1511,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\tentry->type = entry->in_pack_type;\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = oe_size(entry);\n+\t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1895,7 +1897,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -1966,10 +1968,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -1984,7 +1988,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -1997,7 +2001,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2120,11 +2124,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tentry->z_delta_size = size;\n \t\t\tif (entry->z_delta_size == size) {\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0253df6cd4..f1a82bf9ac 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,7 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_DELTA_SIZE_BITS\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -73,9 +74,6 @@ struct object_entry {\n \tstruct pack_idx_entry idx;\n \t/* object uncompressed size _if_ size_valid is true */\n \tuint32_t size_;\n-\n-\t/* XXX 4 bytes hole, try to pack */\n-\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n@@ -84,7 +82,10 @@ struct object_entry {\n \t\t\t\t */\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size;\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_valid:1;\n \tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned type:TYPE_BITS;\n@@ -109,7 +110,7 @@ struct object_entry {\n \t*/\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \n-\t/* size: 88, bit_padding: 1 bits */\n+\t/* size: 80, bit_padding: 1 bits */\n };\n \n struct packing_data {\n@@ -296,4 +297,23 @@ static inline void oe_set_size(struct object_entry *e,\n \te->size_valid = e->size_ == size;\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid =e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(e))\n+\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.16.2.873.g32ff258c87\n\n"},{"id":"341376","messageId":"xmqqpo4c515l.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180308114232.10508-2-pclouds@gmail.com","subject":"Re: [PATCH/RFC v3 01/12] pack-objects: a bit of document about struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-09T22:34:14Z","receivedAt":"2018-03-09T22:34:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> The role of this comment block becomes more important after we shuffle\n> fields around to shrink this struct. It will be much harder to see what\n> field is related to what. This also documents the holes in this struct\n> according to pahole.\n>\n> A couple of notes on shrinking the struct:\n>\n> 1) The reader may notice one thing from this document and the shrinking\n> business. If \"delta\" is NULL, all other delta-related fields should be\n> irrelevant. We could group all these in a separate struct and replace\n> them all with a pointer to this struct (allocated separately).\n>\n> This does not help much though since 85% of objects are deltified\n> (source: linux-2.6.git). The gain is only from non-delta objects, which\n> is not that significant.\n\nOK.\n\n> 2) The field in_pack_offset and idx.offset could be merged. But we need\n> to be very careful. Up until the very last phase (object writing),\n> idx.offset is not used and can hold in_pack_offset. Then idx.offset will\n> be updated with _destination pack's_ offset, not source's. But since we\n> always write delta's bases first, and we only use in_pack_offset in\n> writing phase when we reuse objects, we should be ok?\n\nBy separating the processing in strict phases, I do think the result\nwould be OK, but at the same time, that does smell like an\ninvitation for future bugs.\n\n> +/*\n> + * basic object info\n> + * -----------------\n> + * idx.oid is filled up before delta searching starts. idx.crc32 and\n> + * is only valid after the object is written down and will be used for\n> + * generating the index. idx.offset will be both gradually set and\n> + * used in writing phase (base objects get offset first, then deltas\n> + * refer to them)\n\nHere, I'd feel that \"written out\" somehow would sound more natural\nthan \"written down\", but that is perhaps because I've seen it used\nelsewhere and I am confusing familiarlity with naturalness. In any\ncase, if we mean \"written to the resulting packdata stream\", saying\nthat to be more explicit is probably a good idea. We compute crc32\nand learn the offset for each object as we write them to the result.\n\n> + * If a delta is cached in memory and is compressed, \"delta\" points to\n> + * the data and z_delta_size contains the compressed size. If it's\n\nIsn't it \"delta_data\" (aot \"delta\") that points at the cached delta\ndata?\n\n> + * uncompressed [1], z_delta_size must be zero. delta_size is always\n> + * the uncompressed size and must be valid even if the delta is not\n> + * cached. Delta recreation technically only depends on \"delta\"\n> + * pointer, but delta_size is still used to verify it's the same as\n> + * before.\n> + *\n> + * [1] during try_delta phase we don't bother with compressing because\n> + * the delta could be quickly replaced with a better one.\n> + */\n> struct object_entry {\n> \tstruct pack_idx_entry idx;\n> \tunsigned long size;\t/* uncompressed size */\n> @@ -28,6 +74,7 @@ struct object_entry {\n> \tunsigned tagged:1; /* near the very tip of refs */\n> \tunsigned filled:1; /* assigned write-order */\n> \n> +\t/* XXX 28 bits hole, try to pack */\n> \t/*\n> \t * State flags for depth-first search used for analyzing delta cycles.\n> \t *\n> @@ -40,6 +87,7 @@ struct object_entry {\n> \t\tDFS_DONE\n> \t} dfs_state;\n> \tint depth;\n> +\t/* size: 136, padding: 4 */\n> };\n> \n> struct packing_data {\n"},{"id":"341377","messageId":"xmqqk1uk5076.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180308114232.10508-3-pclouds@gmail.com","subject":"Re: [PATCH/RFC v3 02/12] pack-objects: turn type and in_pack_type to bitfields","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-09T22:54:53Z","receivedAt":"2018-03-09T22:55:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> @@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n> \tentry->depth = 0;\n> \n> \toi.sizep = &entry->size;\n> -\toi.typep = &entry->type;\n> +\toi.typep = &type;\n> \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n> \t\t/*\n> \t\t * We failed to get the info from this pack for some reason;\n> @@ -1580,6 +1586,10 @@ static void drop_reused_delta(struct object_entry *entry)\n> \t\t */\n> \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n> \t\t\t\t\t &entry->size);\n\nThe comment immediately before this pre-context reads as such:\n\n\t\t/*\n\t\t * We failed to get the info from this pack for some reason;\n\t\t * fall back to sha1_object_info, which may find another copy.\n\t\t * And if that fails, the error will be recorded in entry->type\n\t\t * and dealt with in prepare_pack().\n\t\t */\n\nThe rest of the code relies on the ability of entry->type to record\nthe error by storing an invalid (negative) type; otherwise, it cannot\ndetect an error where (1) the entry in _this_ pack was corrupt, and\n(2) we wished to find another copy of the object elsewhere (which\nwould overwrite the negative entry->type we assign here), but we\ndidn't find any.\n\nHow should we propagate the error we found here down the control\nflow in this new code?\n\n> +\t} else {\n> +\t\tif (type < 0)\n> +\t\t\tdie(\"BUG: invalid type %d\", type);\n> +\t\tentry->type = type;\n\nThe BUG() on this side is sensible, as packed_object_info()\nshouldn't report success when it stored negative result in *oi.typep\nanyway.\n\n> \tunsigned char in_pack_header_size;\n> +\tunsigned type:TYPE_BITS;\n> +\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n\n"},{"id":"341379","messageId":"xmqqbmfw4zmg.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180308114232.10508-5-pclouds@gmail.com","subject":"Re: [PATCH/RFC v3 04/12] pack-objects: use bitfield for object_entry::depth","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-09T23:07:19Z","receivedAt":"2018-03-09T23:07:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> +\tMaximum value is 4095.\n> ...\n> +\tif (depth > (1 << OE_DEPTH_BITS))\n> +\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n> +\t\t depth, (1 << OE_DEPTH_BITS));\n> +\n\nDo I see an off-by-one here? Ie.\n\n\tif ((1 << OE_DEPATH_BITS) <= depth)\n\t\tdie(\"depth that is 4096 or deeper is too much\");\n\n"},{"id":"341380","messageId":"xmqq60644yyl.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180308114232.10508-8-pclouds@gmail.com","subject":"Re: [PATCH/RFC v3 07/12] pack-objects: move in_pack out of struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-09T23:21:38Z","receivedAt":"2018-03-09T23:21:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\n> pack. Use an index isntead since the number of packs should be\n> relatively small.\n>\n> This limits the number of packs we can handle to 16k. For now if you hit\n> 16k pack files limit, pack-objects will simply fail [1].\n>\n> This technically saves 7 bytes. But we don't see any of that in\n> practice due to padding. The saving becomes real when we pack this\n> struct tighter later.\n\nSomehow 7 and 16k do not add up.\n\nWe use 8 bytes in the original code, and a solution that potentially\nsaves 7 bytes would use only 1 byte instead of the original 8, which\nwould allow us to index/identify 1<<8 == 256 packs, but for some reason\nwe can handle up to 16k.\n\n> [1] The escape hatch is .keep file to limit the non-kept pack files\n> below 16k limit. Then you can go for another pack-objects run to\n> combine another 16k pack files. Repeat until you're satisfied.\n\n;-)\n\n> +static inline unsigned int oe_add_pack(struct packing_data *pack,\n> +\t\t\t\t struct packed_git *p)\n> +{\n> +\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n> +\t\tdie(_(\"too many packs to handle in one go. \"\n> +\t\t \"Please add .keep files to exclude\\n\"\n> +\t\t \"some pack files and keep the number \"\n> +\t\t \"of non-kept files below %d.\"),\n> +\t\t 1 << OE_IN_PACK_BITS);\n\nOK.\n"},{"id":"341484","messageId":"20180312175111.GA4842@duynguyen.home","threadId":"47943","inReplyTo":"xmqqk1uk5076.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH/RFC v3 02/12] pack-objects: turn type and in_pack_type to bitfields","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-12T17:51:11Z","receivedAt":"2018-03-12T17:51:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 09, 2018 at 02:54:53PM -0800, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n> \n> > @@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n> > \tentry->depth = 0;\n> > \n> > \toi.sizep = &entry->size;\n> > -\toi.typep = &entry->type;\n> > +\toi.typep = &type;\n> > \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n> > \t\t/*\n> > \t\t * We failed to get the info from this pack for some reason;\n> > @@ -1580,6 +1586,10 @@ static void drop_reused_delta(struct object_entry *entry)\n> > \t\t */\n> > \t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n> > \t\t\t\t\t &entry->size);\n> \n> The comment immediately before this pre-context reads as such:\n> \n> \t\t/*\n> \t\t * We failed to get the info from this pack for some reason;\n> \t\t * fall back to sha1_object_info, which may find another copy.\n> \t\t * And if that fails, the error will be recorded in entry->type\n> \t\t * and dealt with in prepare_pack().\n> \t\t */\n> \n> The rest of the code relies on the ability of entry->type to record\n> the error by storing an invalid (negative) type; otherwise, it cannot\n> detect an error where (1) the entry in _this_ pack was corrupt, and\n> (2) we wished to find another copy of the object elsewhere (which\n> would overwrite the negative entry->type we assign here), but we\n> didn't find any.\n> \n> How should we propagate the error we found here down the control\n> flow in this new code?\n\nGood catch! I don't have any magic trick to do this, so I'm adding an\nextra bit to store type validity. Something like this as a fixup patch\n(I'll resend the whole series soon).\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fd217cb51f..f164f1797b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1067,7 +1068,7 @@ static void create_object_entry(const struct object_id *oid,\n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n \tif (type)\n-\t\tentry->type = type;\n+\t\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1433,9 +1434,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1489,7 +1490,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1498,7 +1499,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1521,7 +1522,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1584,12 +1585,10 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * And if that fails, the error will be recorded in entry->type\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n \t} else {\n-\t\tif (type < 0)\n-\t\t\tdie(\"BUG: invalid type %d\", type);\n-\t\tentry->type = type;\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1757,10 +1756,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1836,7 +1837,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2442,11 +2443,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 3e5a89569a..90fbbc9394 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -58,8 +58,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tunsigned type:TYPE_BITS;\n+\tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -122,4 +123,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= 0;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n"},{"id":"341487","messageId":"87d109qh51.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-5-pclouds@gmail.com","subject":"Re: [PATCH v2 4/5] pack-objects: show some progress when counting kept objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-12T18:32:26Z","receivedAt":"2018-03-12T18:32:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 06 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> We only show progress when there are new objects to be packed. But\n> when --keep-pack is specified on the base pack, we will exclude most\n> of objects. This makes 'pack-objects' stay silent for a long time\n> while the counting phase is going.\n>\n> Let's show some progress whenever we visit an object instead. The\n> number of packed objects will be shown after if it's not the same as\n> the number of visited objects.\n>\n> Since the meaning of this number has changed, use another word instead\n> of \"Counting\" to hint about the change.\n\nCan you elaborate on how the meaning has changed? With/without this on\nlinux.git I get:\n\nWith:\n\n Enumerating objects: 5901144, done.\n Getting object details: 100% (5901145/5901145), done.\n Delta compression using up to 8 threads.\n\nWithout:\n\n Counting objects: 5901145, done.\n Delta compression using up to 8 threads.\n\nSo now we're seemingly off-by-one but otherwise doing the same thing?\n\nAs for as user feedback goes we might as well have said \"Reticulating\nsplines\", but I have some bias towards keeping the current \"Counting\nobjects...\" phrasing. We ourselves have other docs referring to it that\naren't changed by this patch, and there's\ne.g. https://githubengineering.com/counting-objects/ and lots of other\n3rd party docs that refer to this.\n"},{"id":"341489","messageId":"87bmftqg1n.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"xmqqd10f20k6.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-12T18:56:04Z","receivedAt":"2018-03-12T18:56:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 07 2018, Junio C. Hamano jotted:\n\n> Duy Nguyen writes:\n>>> But to those who say \"packs larger than this value is too big\" via\n>>> configuration, keeping only the largest of these above-threshold\n>>> packs would look counter-intuitive, wouldn't it, I wonder?\n>>\n>> I think I'll just clarify this in the document. There may be a use\n>> case for keeping multiple large packs, but I don't see it (*). We can\n>> deal with it when it comes.\n>\n> When the project's history grows too much, a large pack that holds\n> its first 10 years of stuff, together with another one that holds\n> its second 20 years of stuff, may both be larger than the threshold\n> and want to be kept. If we pick only the largest one, we would\n> explode the other one and repack together with loose objects.\n>\n> But realistically, those who would want to control the way in which\n> their repository is packed to such a degree are very likely to add\n> \".keep\" files to these two packfiles themselves, so the above would\n> probably not a concern. Perhaps we shouldn't do the \"automatically\n> pick the largest one and exclude from repacking\" when there is a\n> packfile that is marked with \".keep\"?\n\nAs someone who expects to use this (although hopefully in slightly\nmodified form), it's very useful if we can keep the useful semantics in\ngc.* config values without needing some external job finding repos and\ncreating *.keep files to get custom behavior.\n\nE.g. I have the use-case of wanting to set this on servers that I know\nare going to be used for cloning some big repos in user's ~ directory\nmanually, so if I can set something sensible in /etc/gitconfig that's\ngreat, but it sucks a lot more to need to write some cronjob that goes\nhunting for repos in those ~ directories and tweaks *.keep files.\n"},{"id":"341490","messageId":"87a7vdqegi.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-4-pclouds@gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-12T19:30:21Z","receivedAt":"2018-03-12T19:30:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 06 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> pack-objects could be a big memory hog especially on large repos,\n> everybody knows that. The suggestion to stick a .keep file on the\n> giant base pack to avoid this problem is also known for a long time.\n>\n> Let's do the suggestion automatically instead of waiting for people to\n> come to Git mailing list and get the advice. When a certain condition\n> is met, \"gc --auto\" tells \"git repack\" to keep the base pack around.\n> The end result would be two packs instead of one.\n>\n> On linux-2.6.git, valgrind massif reports 1.6GB heap in \"pack all\"\n> case, and 535MB [1] in \"pack all except the base pack\" case. We save\n> roughly 1GB memory by excluding the base pack.\n>\n> gc --auto decides to do this based on an estimation of pack-objects\n> memory usage, which is quite accurate at least for the heap part, and\n> whether that fits in half of system memory (the assumption here is for\n> desktop environment where there are many other applications running).\n>\n> Since the estimation may be inaccurate and that 1/2 threshold is\n> really arbitrary, give the user a finer control over this mechanism:\n> if the largest pack is larger than gc.bigBasePackThreshold, it's kept.\n>\n> PS. A big chunk of the remaining 535MB is the result of pack-objects\n> running rev-list internally. This will be dealt with when we could run\n> rev-list externally. Right now we can't because pack-objects internal\n> rev-list does more regarding unreachable objects, which cannot be done\n> by \"git rev-list\".\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/config.txt | 7 ++\n> Documentation/git-gc.txt | 13 ++++\n> builtin/gc.c | 153 +++++++++++++++++++++++++++++++++++++--\n> builtin/pack-objects.c | 2 +-\n> config.mak.uname | 1 +\n> git-compat-util.h | 4 +\n> pack-objects.h | 2 +\n> t/t6500-gc.sh | 29 ++++++++\n> 8 files changed, 204 insertions(+), 7 deletions(-)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index f57e9cf10c..120cf6bac9 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1549,6 +1549,13 @@ gc.autoDetach::\n> \tMake `git gc --auto` return immediately and run in background\n> \tif the system supports it. Default is true.\n>\n> +gc.bigBasePackThreshold::\n> +\tMake `git gc --auto` only enable `--keep-base-pack` when the\n> +\tbase pack's size is larger than this limit (in bytes).\n> +\tDefaults to zero, which disables this check and lets\n> +\t`git gc --auto` determine when to enable `--keep-base-pack`\n> +\tbased on memory usage.\n> +\n\nI'm really keen to use this (and would be happy to apply a patch on\ntop), but want to get your thoughts first, see also my just-sent\n87bmftqg1n.fsf@evledraar.gmail.com\n(https://public-inbox.org/git/87bmftqg1n.fsf@evledraar.gmail.com/).\n\nThe thing I'd like to change is that the underlying --keep-pack= takes a\nlist of paths (good!), but then I think this patch needlessly\ncomplicates things by talking about \"base packs\" and having the\nimplementation limitation that we only ever pass one --keep-pack down to\npack-objects (bad!).\n\nWhy don't we instead just have a gc.* variable that you can set to some\nsize of pack that we'll always implicitly *.keep? That way I can\ne.g. clone a 5GB pack and set the limit to 2GB, then keep adding new\ncontent per the rules of gc.autoPackLimit, until I finally create a\nlarger than 2GB pack, at that point I'll have 5GB, 2GB, and some smaller\npacks and loose objects.\n\nWe already have pack.packSizeLimit, perhaps we could call this\ne.g. gc.keepPacksSize=2GB?\n\nOr is there a use-case for still having the concept of a \"base\" pack? Is\nit magic in some way? Maybe I'm missing something but I don't see why,\nwe can just stop thinking about whether some one pack is larger than X,\nand consider all packs larger than X specially.\n\nBut if we do maybe an extra gc.keepBasePack=true?\n\nFinally I wonder if there should be something equivalent to\ngc.autoPackLimit for this. I.e. with my proposed semantics above it's\npossible that we end up growing forever, i.e. I could have 1000 2GB\npacks and then 50 very small packs per gc.autoPackLimit.\n\nMaybe we need a gc.keepPackLimit=100 to deal with that, then e.g. if\ngc.keepPacksSize=2GB is set and we have 101 >= 2GB packs, we'd pick the\ntwo smallest one and not issue a --keep-pack for those, although then\nmaybe our memory use would spike past the limit.\n\nI don't know, maybe we can leave that for later, but I'm quite keen to\nturn the top-level config variable into something that just considers\nsize instead of \"base\" if possible, and it seems we're >95% of the way\nto that already with this patch.\n\nFinally, I don't like the way the current implementation conflates a\n\"size\" variable with auto detecting the size from memory, leaving no way\nto fallback to the auto-detection if you set it manually.\n\nI think we should split out the auto-memory behavior into another\nvariable, and also make the currently hardcoded 50% of memory\nconfigurable.\n\nThat way you could e.g. say you'd always like to keep 2GB packs, but if\nyou happen to have ended up with a 1GB pack and it's time to repack, and\nyou only have 500MB free memory on that system, it would keep the 1GB\none until such time as we have more memory.\n\nActually maybe that should be a \"if we're that low on memory, forget\nabout GC for now\" config, but urgh, there's a lot of potential\ncomplexity to be handled here...\n"},{"id":"341491","messageId":"CAPc5daUcA8MVKH515gvDzUXBThq=8+EXouOQ2zxZGmp5YH5nfw@mail.gmail.com","threadId":"47943","inReplyTo":"87bmftqg1n.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-12T21:16:17Z","receivedAt":"2018-03-12T21:16:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Mon, Mar 12, 2018 at 11:56 AM, Ævar Arnfjörð Bjarmason\n wrote:\n> As someone who expects to use this (although hopefully in slightly\n> modified form), it's very useful if we can keep the useful semantics in\n> gc.* config values without needing some external job finding repos and\n> creating *.keep files to get custom behavior.\n>\n> E.g. I have the use-case of wanting to set this on servers that I know\n> are going to be used for cloning some big repos in user's ~ directory\n> manually, so if I can set something sensible in /etc/gitconfig that's\n> great, but it sucks a lot more to need to write some cronjob that goes\n> hunting for repos in those ~ directories and tweaks *.keep files.\n\nYeah, but that is exactly what I suggested, no? That is, if you don't do any\nspecific marking to describe _which_ ones need to be kept, this new thing\nwould kick in and pick the largest one and repack all others. If you choose\nto want more control, on the other hand, you can mark those packs you\nwould want to keep, and this mechanism will not kick in to countermand\nyour explicit settings done via those .keep files.\n"},{"id":"341496","messageId":"877eqhq7ha.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"CAPc5daUcA8MVKH515gvDzUXBThq=8+EXouOQ2zxZGmp5YH5nfw@mail.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-12T22:01:05Z","receivedAt":"2018-03-12T22:01:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 12 2018, Junio C. Hamano jotted:\n\n> On Mon, Mar 12, 2018 at 11:56 AM, Ævar Arnfjörð Bjarmason\n> wrote:\n>> As someone who expects to use this (although hopefully in slightly\n>> modified form), it's very useful if we can keep the useful semantics in\n>> gc.* config values without needing some external job finding repos and\n>> creating *.keep files to get custom behavior.\n>>\n>> E.g. I have the use-case of wanting to set this on servers that I know\n>> are going to be used for cloning some big repos in user's ~ directory\n>> manually, so if I can set something sensible in /etc/gitconfig that's\n>> great, but it sucks a lot more to need to write some cronjob that goes\n>> hunting for repos in those ~ directories and tweaks *.keep files.\n>\n> Yeah, but that is exactly what I suggested, no? That is, if you don't do any\n> specific marking to describe _which_ ones need to be kept, this new thing\n> would kick in and pick the largest one and repack all others. If you choose\n> to want more control, on the other hand, you can mark those packs you\n> would want to keep, and this mechanism will not kick in to countermand\n> your explicit settings done via those .keep files.\n\nYes, this configurable mechanism as it stands only needs /etc/gitconfig.\n\nWhat I was pointing out in this mail is that we really should get the\nadvanced use-cases right as well (see my\n87a7vdqegi.fsf@evledraar.gmail.com for details) via the config, because\nit's a pain to cross the chasm between setting config centrally on the\none hand, and needing to track down .git's in arbitrary locations on the\nFS (you may not have cloned them yourself) to set *.keep flags.\n\nDoubly so if the machines in questions are just the laptops of some\ndevelopers. It's relatively easy to tell them \"we work with git repos,\nrun this git config commands\", not so easy to have them install & keep\nup-to-date some arbitrary cronjob that needs to hunt down their repos\nand set *.keep flags.\n"},{"id":"341638","messageId":"xmqqy3iuzl3i.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180308114232.10508-9-pclouds@gmail.com","subject":"Re: [PATCH/RFC v3 08/12] pack-objects: refer to delta objects by index instead of pointer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-14T16:18:57Z","receivedAt":"2018-03-14T16:19:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Notice that packing_data::nr_objects is uint32_t, we could only handle\n> maximum 4G objects and can address all of them with an uint32_t. If we\n> use a pointer here, we waste 4 bytes on 64 bit architecture.\n\nSome things are left unsaid or left unclear and make readers stutter\na bit while reading this paragraph. We can address them with\nuint32_t only because we happen to have a linear array of all\nobjects involved already, i.e. the pack->objects[] array. The\nreaders are forced to rephrase the above in their mind\n\n\t... and each of them can be identified with an uint32_t.\n\tBecause we have all of these objects in pack->objects[], we\n\tcan replace the \"delta\" field in each object entry that\n\tpoints at its delta base object with uint32_t index into\n\tthis array to save memory (on 64-bit arch, 8-byte pointer\n\tgets shrunk to 4-byte uint).\n\nor something like that before understanding why this is a valid\nmemory footprint optimization.\n\n"},{"id":"341763","messageId":"CACsJy8AqQbX5Mkz=Gz=G79sp0mjV5rsyeo2X=pSO9HCuDbh1sw@mail.gmail.com","threadId":"47943","inReplyTo":"87bmftqg1n.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-15T16:48:09Z","receivedAt":"2018-03-15T16:48:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 12, 2018 at 7:56 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Wed, Mar 07 2018, Junio C. Hamano jotted:\n>\n>> Duy Nguyen writes:\n>>>> But to those who say \"packs larger than this value is too big\" via\n>>>> configuration, keeping only the largest of these above-threshold\n>>>> packs would look counter-intuitive, wouldn't it, I wonder?\n>>>\n>>> I think I'll just clarify this in the document. There may be a use\n>>> case for keeping multiple large packs, but I don't see it (*). We can\n>>> deal with it when it comes.\n>>\n>> When the project's history grows too much, a large pack that holds\n>> its first 10 years of stuff, together with another one that holds\n>> its second 20 years of stuff, may both be larger than the threshold\n>> and want to be kept. If we pick only the largest one, we would\n>> explode the other one and repack together with loose objects.\n>>\n>> But realistically, those who would want to control the way in which\n>> their repository is packed to such a degree are very likely to add\n>> \".keep\" files to these two packfiles themselves, so the above would\n>> probably not a concern. Perhaps we shouldn't do the \"automatically\n>> pick the largest one and exclude from repacking\" when there is a\n>> packfile that is marked with \".keep\"?\n>\n> As someone who expects to use this (although hopefully in slightly\n> modified form), it's very useful if we can keep the useful semantics in\n> gc.* config values without needing some external job finding repos andis is\n> creating *.keep files to get custom behavior.\n>\n> E.g. I have the use-case of wanting to set this on servers that I know\n> are going to be used for cloning some big repos in user's ~ directory\n> manually, so if I can set something sensible in /etc/gitconfig that's\n> great, but it sucks a lot more to need to write some cronjob that goes\n> hunting for repos in those ~ directories and tweaks *.keep files.\n\nIf this is about .gc.bigBasePackThreshold keeping all packs larger\nthan the threshold, then yes it will be so in the reroll.\n-- \nDuy\n"},{"id":"341766","messageId":"CACsJy8DZGa8v6z6c3SUjENMd4h6Qi55iPfsT2c5-cejqv5jA2Q@mail.gmail.com","threadId":"47943","inReplyTo":"87a7vdqegi.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-15T17:00:35Z","receivedAt":"2018-03-15T17:01:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 12, 2018 at 8:30 PM, Ævar Arnfjörð Bjarmason\n wrote:\n> We already have pack.packSizeLimit, perhaps we could call this\n> e.g. gc.keepPacksSize=2GB?\n\nI'm OK either way. The \"base pack\" concept comes from the\n\"--keep-base-pack\" option where we do keep _one_ base pack. But gc\nconfig var has a slightly different semantics when it can keep\nmultiple packs.\n\n> Finally I wonder if there should be something equivalent to\n> gc.autoPackLimit for this. I.e. with my proposed semantics above it's\n> possible that we end up growing forever, i.e. I could have 1000 2GB\n> packs and then 50 very small packs per gc.autoPackLimit.\n>\n> Maybe we need a gc.keepPackLimit=100 to deal with that, then e.g. if\n> gc.keepPacksSize=2GB is set and we have 101 >= 2GB packs, we'd pick the\n> two smallest one and not issue a --keep-pack for those, although then\n> maybe our memory use would spike past the limit.\n>\n> I don't know, maybe we can leave that for later, but I'm quite keen to\n> turn the top-level config variable into something that just considers\n> size instead of \"base\" if possible, and it seems we're >95% of the way\n> to that already with this patch.\n\nAt least I will try to ignore gc.keepPacksSize if all packs are kept\nbecause of it. That repack run will hurt. But then we're back to one\ngiant pack and plenty of small packs that will take some time to grow\nup to 2GB again.\n\n> Finally, I don't like the way the current implementation conflates a\n> \"size\" variable with auto detecting the size from memory, leaving no way\n> to fallback to the auto-detection if you set it manually.\n>\n> I think we should split out the auto-memory behavior into another\n> variable, and also make the currently hardcoded 50% of memory\n> configurable.\n>\n> That way you could e.g. say you'd always like to keep 2GB packs, but if\n> you happen to have ended up with a 1GB pack and it's time to repack, and\n> you only have 500MB free memory on that system, it would keep the 1GB\n> one until such time as we have more memory.\n\nI don't calculate based on free memory (it's tricky to get that right\non linux) but physical memory. If you don't have enough memory\naccording to this formula, you won't until you add more memory sticks.\n\n>\n> Actually maybe that should be a \"if we're that low on memory, forget\n> about GC for now\" config, but urgh, there's a lot of potential\n> complexity to be handled here...\n\nYeah I think what you want is a hook. You can make custom rules then.\nWe already have pre-auto-gc hook and could pretty much do what you\nwant without pack-objects memory estimation. But if you want it, maybe\nwe can export the info to the hook somehow.\n-- \nDuy\n"},{"id":"341825","messageId":"87muz9du0w.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"CACsJy8DZGa8v6z6c3SUjENMd4h6Qi55iPfsT2c5-cejqv5jA2Q@mail.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-15T19:21:35Z","receivedAt":"2018-03-15T19:21:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 15 2018, Duy Nguyen jotted:\n\n> On Mon, Mar 12, 2018 at 8:30 PM, Ævar Arnfjörð Bjarmason\n> wrote:\n>> We already have pack.packSizeLimit, perhaps we could call this\n>> e.g. gc.keepPacksSize=2GB?\n>\n> I'm OK either way. The \"base pack\" concept comes from the\n> \"--keep-base-pack\" option where we do keep _one_ base pack. But gc\n> config var has a slightly different semantics when it can keep\n> multiple packs.\n\nI see, yeah it would be great to generalize it to N packs.\n\n>> Finally I wonder if there should be something equivalent to\n>> gc.autoPackLimit for this. I.e. with my proposed semantics above it's\n>> possible that we end up growing forever, i.e. I could have 1000 2GB\n>> packs and then 50 very small packs per gc.autoPackLimit.\n>>\n>> Maybe we need a gc.keepPackLimit=100 to deal with that, then e.g. if\n>> gc.keepPacksSize=2GB is set and we have 101 >= 2GB packs, we'd pick the\n>> two smallest one and not issue a --keep-pack for those, although then\n>> maybe our memory use would spike past the limit.\n>>\n>> I don't know, maybe we can leave that for later, but I'm quite keen to\n>> turn the top-level config variable into something that just considers\n>> size instead of \"base\" if possible, and it seems we're >95% of the way\n>> to that already with this patch.\n>\n> At least I will try to ignore gc.keepPacksSize if all packs are kept\n> because of it. That repack run will hurt. But then we're back to one\n> giant pack and plenty of small packs that will take some time to grow\n> up to 2GB again.\n\nI think that semantic really should have its own option. The usefulness\nof this is significantly diminished if it's not a guarantee on the\nresource use of git-gc.\n\nConsider a very large repo where we clone and get a 4GB pack. Then as\ntime goes on we end up with lots of loose objects and small packs from\npulling, and eventually end up with say 4GB + 2x 500MB packs (if our\nlimit is 500MB).\n\nIf I understand what you're saying correctly if we ever match the gc\n--auto requirements because we have *just* the big packs and then a\nbunch of loose objects (say we rebased a lot) then we'll try to create a\ngiant 5GB pack (+ loose objects).\n\n>> Finally, I don't like the way the current implementation conflates a\n>> \"size\" variable with auto detecting the size from memory, leaving no way\n>> to fallback to the auto-detection if you set it manually.\n>>\n>> I think we should split out the auto-memory behavior into another\n>> variable, and also make the currently hardcoded 50% of memory\n>> configurable.\n>>\n>> That way you could e.g. say you'd always like to keep 2GB packs, but if\n>> you happen to have ended up with a 1GB pack and it's time to repack, and\n>> you only have 500MB free memory on that system, it would keep the 1GB\n>> one until such time as we have more memory.\n>\n> I don't calculate based on free memory (it's tricky to get that right\n> on linux) but physical memory. If you don't have enough memory\n> according to this formula, you won't until you add more memory sticks.\n\nAh, thanks for the clarification.\n\n>>\n>> Actually maybe that should be a \"if we're that low on memory, forget\n>> about GC for now\" config, but urgh, there's a lot of potential\n>> complexity to be handled here...\n>\n> Yeah I think what you want is a hook. You can make custom rules then.\n> We already have pre-auto-gc hook and could pretty much do what you\n> want without pack-objects memory estimation. But if you want it, maybe\n> we can export the info to the hook somehow.\n\nI can do away with that particular thing, but I'd really like to do\nwithout the hook. I can automate it on some machines, but then we also\nhave un-managed laptops run by users who clone big repos. It's much\neasier to tell them to set a few git config variables than have them\ninstall & keep some hook up-to-date.\n"},{"id":"341883","messageId":"CACsJy8DCSQCAsLKx8PepyT_MbYSBuaArQc5CWBdxBM5svry56Q@mail.gmail.com","threadId":"47943","inReplyTo":"87muz9du0w.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T17:47:44Z","receivedAt":"2018-03-16T17:48:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 15, 2018 at 8:21 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Thu, Mar 15 2018, Duy Nguyen jotted:\n>\n>> On Mon, Mar 12, 2018 at 8:30 PM, Ævar Arnfjörð Bjarmason\n>> wrote:\n>>> We already have pack.packSizeLimit, perhaps we could call this\n>>> e.g. gc.keepPacksSize=2GB?\n>>\n>> I'm OK either way. The \"base pack\" concept comes from the\n>> \"--keep-base-pack\" option where we do keep _one_ base pack. But gc\n>> config var has a slightly different semantics when it can keep\n>> multiple packs.\n>\n> I see, yeah it would be great to generalize it to N packs.\n>\n>>> Finally I wonder if there should be something equivalent to\n>>> gc.autoPackLimit for this. I.e. with my proposed semantics above it's\n>>> possible that we end up growing forever, i.e. I could have 1000 2GB\n>>> packs and then 50 very small packs per gc.autoPackLimit.\n>>>\n>>> Maybe we need a gc.keepPackLimit=100 to deal with that, then e.g. if\n>>> gc.keepPacksSize=2GB is set and we have 101 >= 2GB packs, we'd pick the\n>>> two smallest one and not issue a --keep-pack for those, although then\n>>> maybe our memory use would spike past the limit.\n>>>\n>>> I don't know, maybe we can leave that for later, but I'm quite keen to\n>>> turn the top-level config variable into something that just considers\n>>> size instead of \"base\" if possible, and it seems we're >95% of the way\n>>> to that already with this patch.\n>>\n>> At least I will try to ignore gc.keepPacksSize if all packs are kept\n>> because of it. That repack run will hurt. But then we're back to one\n>> giant pack and plenty of small packs that will take some time to grow\n>> up to 2GB again.\n>\n> I think that semantic really should have its own option. The usefulness\n> of this is significantly diminished if it's not a guarantee on the\n> resource use of git-gc.\n>\n> Consider a very large repo where we clone and get a 4GB pack. Then as\n> time goes on we end up with lots of loose objects and small packs from\n> pulling, and eventually end up with say 4GB + 2x 500MB packs (if our\n> limit is 500MB).\n>\n> If I understand what you're saying correctly if we ever match the gc\n> --auto requirements because we have *just* the big packs and then a\n> bunch of loose objects (say we rebased a lot) then we'll try to create a\n> giant 5GB pack (+ loose objects).\n\nYes. There isn't a simple and easy solution here and I consider\npacking (even if it's expensive) to regain performance is better than\nnot packing at all. I could tweak that a bit by keeping the largest\npack out (so we have to packs in the end). After a long long long time\nwhen your second pack gets to 5 GB, then we hit the most expensive\nrepack. But that should be ok for now, I guess.\n\nI think this repack strategy was discussed here at some point in the\npast by Gerrit guys. Their goal was to reduce I/O, I believe. A\nperfect solution probably could be found, but I don't want to hold\nthis series back until it's found and I don't want to introduce a\nzillion config knobs that become useless later on when the perfect\nsolution is found.\n\n>>> Actually maybe that should be a \"if we're that low on memory, forget\n>>> about GC for now\" config, but urgh, there's a lot of potential\n>>> complexity to be handled here...\n>>\n>> Yeah I think what you want is a hook. You can make custom rules then.\n>> We already have pre-auto-gc hook and could pretty much do what you\n>> want without pack-objects memory estimation. But if you want it, maybe\n>> we can export the info to the hook somehow.\n>\n> I can do away with that particular thing, but I'd really like to do\n> without the hook. I can automate it on some machines, but then we also\n> have un-managed laptops run by users who clone big repos. It's much\n> easier to tell them to set a few git config variables than have them\n> install & keep some hook up-to-date.\n\nThat sounds like we need a mechanism to push hooks (and config stuff)\nautomatically from clone source. I think this topic was touched in the\nsummit? I don't object adding new config but we need to figure out\nwhat we need, and from this thread I think there are too many \"I don't\nknow\" to settle on a solution.\n-- \nDuy\n"},{"id":"341889","messageId":"20180316183200.31014-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 01/11] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:50Z","receivedAt":"2018-03-16T18:32:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..85345a4af1 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,50 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 and\n+ * is only valid after the object is written out and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size is not\n+ * cached (ie. raw data in a pack) but available via revindex.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n+ * the location of the object in the source pack, with or without\n+ * header.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341890","messageId":"20180316183200.31014-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180308114232.10508-1-pclouds@gmail.com","subject":"[PATCH v4 00/11] nd/pack-objects-pack-struct updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:49Z","receivedAt":"2018-03-16T18:32:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The most important change in v4 is it fixes a case where I failed to\npropagate an error condition to later code in 02/11. This results in\nnew wrappers, oe_type() and oe_set_type(). This also reveals another\nextra object type, OBJ_NONE, that's also used by pack-objects.\n\nOther changes are comments fixes, commit messages fixes, off-by-one\nbugs. No more saving compared to v3.\n\nI also changed my approach a bit. I stop trying to make struct\nreduction visible at every patch. All these patches shrink some field\neven if the struct size is the same. The reordering and packing\nhappens at the last patch.\n\nI'm not super happy that many corner cases of my changes are not\ncovered by the test suite. In many cases it's very hard or expensive\nto create the right error condition. If only this code is part of\nlibgit.a and I could write C unit tests for it...\n\nInterdiff\n\n-- 8< --\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 0f65e0f243..c388d87c3e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -275,7 +275,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t oe_size_greater_than(entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -381,7 +381,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -491,11 +491,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -716,8 +717,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -726,7 +727,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1083,8 +1084,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1453,9 +1453,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1509,7 +1509,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n \t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n@@ -1518,7 +1518,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\tunsigned long size;\n \n \t\t\tsize = get_size_from_delta(p, &w_curs,\n@@ -1544,14 +1544,15 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n \t * as a preferred base. Doing so can result in a larger\n \t * pack file, but the transfer will still take place.\n \t */\n-\toe_set_size(entry, size);\n+\tif (entry->type_valid)\n+\t\toe_set_size(entry, size);\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1613,15 +1614,12 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * And if that fails, the error will be recorded in entry->type\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &size);\n-\t\toe_set_size(entry, size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &size));\n \t} else {\n-\t\tif (type < 0)\n-\t\t\tdie(\"BUG: invalid type %d\", type);\n-\t\tentry->type = type;\n-\t\toe_set_size(entry, size);\n+\t\toe_set_type(entry, type);\n \t}\n+\toe_set_size(entry, size);\n }\n \n /*\n@@ -1788,12 +1786,14 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \tunsigned long a_size = oe_size(a);\n \tunsigned long b_size = oe_size(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1869,7 +1869,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2484,11 +2484,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2597,7 +2597,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \n@@ -3110,7 +3110,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n-\tif (depth > (1 << OE_DEPTH_BITS))\n+\tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS));\n \tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 256a63f892..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -66,12 +66,12 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \n \t\toe_set_in_pack_pos(to_pack, entry, i);\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -100,7 +100,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 1a159aba37..0fa0c83294 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -28,7 +28,7 @@ enum dfs_state {\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 and\n- * is only valid after the object is written down and will be used for\n+ * is only valid after the object is written out and will be used for\n * generating the index. idx.offset will be both gradually set and\n * used in writing phase (base objects get offset first, then deltas\n * refer to them)\n@@ -59,8 +59,8 @@ enum dfs_state {\n * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n * at object writing phase in case the delta is not cached.\n *\n- * If a delta is cached in memory and is compressed delta points to\n- * the data and z_delta_size contains the compressed size. If it's\n+ * If a delta is cached in memory and is compressed delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n * uncompressed [1], z_delta_size must be zero. delta_size is always\n * the uncompressed size and must be valid even if the delta is not\n * cached.\n@@ -70,23 +70,22 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\t/* object uncompressed size _if_ size_valid is true */\n-\tuint32_t size_;\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n+\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\t/* object uncompressed size _if_ size_valid is true */\n-\tuint32_t size;\n-\tuint32_t delta_size_:OE_DELTA_SIZE_BITS;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n-\tunsigned char in_pack_header_size; /* note: spare bits available! */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n-\tunsigned type:TYPE_BITS;\n+\tunsigned size_valid:1;\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n+\tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -94,21 +93,13 @@ struct object_entry {\n \t\t\t\t * objects against.\n \t\t\t\t */\n \tunsigned no_try_delta:1;\n+\tunsigned char in_pack_header_size;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n-\tunsigned size_valid:1;\n-\n-\t/* XXX 8 bits hole, try to pack */\n-\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n-\t/*\n-\t * if delta_data contains a compressed delta, this contains\n-\t * the compressed length\n-\t*/\n-\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \n-\t/* size: 80, bit_padding: 1 bits */\n+\t/* size: 80, bit_padding: 16 bits */\n };\n \n struct packing_data {\n@@ -151,6 +142,21 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n \t\t\t\t\t const struct object_entry *e)\n {\n-- 8< --\n\nNguyễn Thái Ngọc Duy (11):\n pack-objects: a bit of document about struct object_entry\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: shrink z_delta_size field in struct object_entry\n pack-objects: shrink size field in struct object_entry\n pack-objects: shrink delta_size field in struct object_entry\n pack-objects.h: reorder members to shrink struct object_entry\n\n Documentation/config.txt | 4 +-\n Documentation/git-pack-objects.txt | 13 +-\n Documentation/git-repack.txt | 4 +-\n builtin/pack-objects.c | 309 +++++++++++++++++------------\n cache.h | 3 +\n object.h | 1 -\n pack-bitmap-write.c | 14 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.h | 294 ++++++++++++++++++++++++---\n 10 files changed, 488 insertions(+), 160 deletions(-)\n\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341891","messageId":"20180316183200.31014-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 02/11] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:51Z","receivedAt":"2018-03-16T18:32:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"An extra field type_valid is added to carry the equivalent of OBJ_BAD\nin the original \"type\" field. in_pack_type always contains a valid\ntype so we only need 3 bits for it.\n\nA note about accepting OBJ_NONE as \"valid\" type. The function\nread_object_list_from_stdin() can pass this value [1] and it\neventually calls create_object_entry() where current code skip setting\n\"type\" field if the incoming type is zero. This does not have any bad\nside effects because \"type\" field should be memset()'d anyway.\n\nBut since we also need to set type_valid now, skipping oe_set_type()\nleaves type_valid zero/false, which will make oe_type() return\nOBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in\nprepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger\n\n fatal: unable to get type of object ...\n\nAccepting OBJ_NONE [2] does sound wrong, but this is how it is has\nbeen for a very long time and I haven't time to dig in further.\n\n[1] See 5c49c11686 (pack-objects: better check_object() performances -\n 2007-04-16)\n\n[2] 21666f1aae (convert object type handling from a string to a number\n - 2007-02-26)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 58 +++++++++++++++++++++++++-----------------\n cache.h | 2 ++\n object.h | 1 -\n pack-bitmap-write.c | 6 ++---\n pack-objects.h | 20 +++++++++++++--\n 5 files changed, 57 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..13f6a44fb2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1066,8 +1067,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1428,9 +1433,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1484,7 +1489,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1493,7 +1498,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1516,7 +1521,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n@@ -1578,8 +1584,10 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * And if that fails, the error will be recorded in entry->type\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n+\t} else {\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1747,10 +1755,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1826,7 +1836,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2432,11 +2442,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2545,7 +2555,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..fd11f08940 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \n \t\tentry->in_pack_pos = i;\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -98,7 +98,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 85345a4af1..38d3ff167f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -58,8 +58,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n+\tunsigned type_:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -122,4 +123,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341892","messageId":"20180316183200.31014-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:53Z","receivedAt":"2018-03-16T18:33:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Because of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 5 ++---\n 5 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 09f8b4ef3e..668eaf8cd7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth >= (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS));\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 2bb1732098..50908d1f2d 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -88,9 +89,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\tint depth;\n-\n+\tunsigned depth:OE_DEPTH_BITS;\n };\n \n struct packing_data {\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341893","messageId":"20180316183200.31014-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 05/11] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:54Z","receivedAt":"2018-03-16T18:33:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 16 +++++++++++++++-\n 5 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 668eaf8cd7..b281487b96 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -879,7 +879,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fd11f08940..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 50908d1f2d..dae160e7c2 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -78,7 +78,6 @@ struct object_entry {\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -98,6 +97,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -143,4 +144,17 @@ static inline void oe_set_type(struct object_entry *e,\n \te->type_ = (unsigned)type;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341894","messageId":"20180316183200.31014-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 03/11] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:52Z","receivedAt":"2018-03-16T18:33:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 28 +++++++++++++++++-----------\n 2 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 13f6a44fb2..09f8b4ef3e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 38d3ff167f..2bb1732098 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -72,19 +87,10 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n+\n };\n \n struct packing_data {\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341895","messageId":"20180316183200.31014-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 08/11] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:57Z","receivedAt":"2018-03-16T18:33:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 16 bits, so you can only cache 65kb deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 22 ++++++++++++++++------\n pack-objects.h | 3 ++-\n 3 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex cdbad57082..9a0962cf31 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2105,12 +2105,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tentry->z_delta_size = size;\n+\t\t\tif (entry->z_delta_size == size) {\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3089,6 +3096,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS));\n+\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n+\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n+\t\t 1 << OE_Z_DELTA_BITS);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 7f32de2a35..a66c37e35a 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n+#define OE_Z_DELTA_BITS\t\t16\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -78,7 +79,7 @@ struct object_entry {\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341896","messageId":"20180316183200.31014-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:32:00Z","receivedAt":"2018-03-16T18:33:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Previous patches leave lots of holes and padding in this struct. This\npatch reorders the members and shrinks the struct down to 80 bytes\n(from 136 bytes, before any field shrinking is done) with 16 bits to\nspare (and a couple more in in_pack_header_size when we really run out\nof bits).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 17 +++++++++--------\n 1 file changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex f430d938c6..0fa0c83294 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -70,35 +70,36 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\t/* object uncompressed size _if_ size_valid is true */\n-\tuint32_t size_;\n-\tunsigned size_valid:1;\n-\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n+\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tunsigned size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n-\tunsigned type_valid:1;\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n \tunsigned no_try_delta:1;\n+\tunsigned char in_pack_header_size;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n+\n+\t/* size: 80, bit_padding: 16 bits */\n };\n \n struct packing_data {\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341897","messageId":"20180316183200.31014-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 07/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:56Z","receivedAt":"2018-03-16T18:33:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These delta pointers always point to elements in the objects[] array\nin packing_data struct. We can only hold maximum 4GB of those objects\nbecause the array length, nr_objects, is uint32_t. We could use\nuint32_t indexes to address these elements instead of pointers. On\n64-bit architecture (8 bytes per pointer) this would save 4 bytes per\npointer.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 67 ++++++++++++++++++++++--\n 2 files changed, 124 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca993e55dd..cdbad57082 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -489,7 +495,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -541,12 +547,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -608,34 +614,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -646,7 +652,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -661,8 +667,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -672,11 +678,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1498,10 +1504,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1572,17 +1578,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1622,7 +1630,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1647,7 +1655,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1670,7 +1678,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1685,7 +1693,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1870,7 +1878,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1946,7 +1954,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1975,7 +1983,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1984,13 +1992,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2059,7 +2067,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2109,7 +2117,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2117,7 +2125,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2438,7 +2446,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 9bcb5946e5..7f32de2a35 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -71,11 +71,11 @@ struct object_entry {\n \tunsigned long size;\t/* uncompressed size */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -199,4 +199,61 @@ static inline void oe_set_in_pack(struct object_entry *e,\n \n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341898","messageId":"20180316183200.31014-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 10/11] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:59Z","receivedAt":"2018-03-16T18:33:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n31 bits with one overflow bit.\n\nIf we find an existing delta larger than 2GB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 24 ++++++++++++++----------\n pack-objects.h | 23 ++++++++++++++++++++++-\n 2 files changed, 36 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 14aa4acd50..c388d87c3e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,10 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -140,7 +142,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -291,14 +293,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1509,7 +1511,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = oe_size(entry);\n+\t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1895,7 +1897,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -1966,10 +1968,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -1984,7 +1988,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -1997,7 +2001,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2120,11 +2124,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tentry->z_delta_size = size;\n \t\t\tif (entry->z_delta_size == size) {\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 5c7e15ca92..f430d938c6 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,7 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_DELTA_SIZE_BITS\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -80,7 +81,8 @@ struct object_entry {\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tuint32_t delta_size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -298,4 +300,23 @@ static inline void oe_set_size(struct object_entry *e,\n \te->size_valid = e->size_ == size;\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid =e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(e))\n+\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341899","messageId":"20180316183200.31014-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 06/11] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:55Z","receivedAt":"2018-03-16T18:33:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index instead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 16k. For now if you hit\n16k pack files limit, pack-objects will simply fail [1].\n\n[1] The escape hatch is .keep file to limit the non-kept pack files\n below 16k limit. Then you can go for another pack-objects run to\n combine another 16k pack files. Repeat until you're satisfied.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++++\n builtin/pack-objects.c | 40 +++++++++++++++++----------\n cache.h | 1 +\n pack-objects.h | 44 +++++++++++++++++++++++++++++-\n 4 files changed, 79 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 3503c9e3e6..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b281487b96..ca993e55dd 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -1025,7 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1048,11 +1050,16 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index)\n+\t\toe_add_pack(&to_pack, *found_pack);\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1074,7 +1081,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1406,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1542,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1587,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1857,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -3191,6 +3200,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\t/* make sure IN_PACK(0) return NULL */\n+\toe_add_pack(&to_pack, NULL);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex dae160e7c2..9bcb5946e5 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t14\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -18,6 +19,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 and\n@@ -64,7 +69,7 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -99,6 +104,8 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -157,4 +164,39 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline unsigned int oe_add_pack(struct packing_data *pack,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t \"Please add .keep files to exclude\\n\"\n+\t\t \"some pack files and keep the number \"\n+\t\t \"of non-kept files below %d.\"),\n+\t\t 1 << OE_IN_PACK_BITS);\n+\tif (p) {\n+\t\tif (p->index > 0)\n+\t\t\tdie(\"BUG: this packed is already indexed\");\n+\t\tp->index = pack->in_pack_count;\n+\t}\n+\tpack->in_pack[pack->in_pack_count] = p;\n+\treturn pack->in_pack_count++;\n+}\n+\n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack[e->in_pack_idx];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (p->index <= 0)\n+\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t \"instead of having non-positive index\");\n+\te->in_pack_idx = p->index;\n+\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341900","messageId":"20180316183200.31014-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v4 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T18:31:58Z","receivedAt":"2018-03-16T18:33:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressd object is larger than 4GB\n(partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size\nis smaller than this limit.\n\nShrink size field down to 32 bits [1] and one overflow bit. If the size\nis too large, we read it back from disk.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nA small note about the conditional oe_set_size() in\ncheck_object(). Technically if we don't get a valid type, it's not\nwrong if we set uninitialized value \"size\" (we don't pre-initialize\nthis and sha1_object_info will not assign anything when it fails to\nget the info).\n\nThis how changes the writing code path slightly which emits different\nerror messages (either way we die). One of our tests in t5530 depends\non this specific error message. Let's just keep the test as-is and\nplay safe by not assigning random value. That might trigger valgrind\nanyway.\n\n[1] it's actually already 32 bits on Windows\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 49 ++++++++++++++++++++++++++----------------\n pack-objects.h | 43 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 73 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9a0962cf31..14aa4acd50 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -274,7 +274,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +384,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = oe_size(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +407,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1412,6 +1413,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1431,13 +1434,14 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n+\t\toe_set_size(entry, size);\n \n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n@@ -1505,7 +1509,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = oe_size(entry);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1513,14 +1517,17 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = get_size_from_delta(p, &w_curs,\n+\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n-\t\t\tif (entry->size == 0)\n+\t\t\toe_set_size(entry, size);\n+\t\t\tif (oe_size_less_than(entry, 1))\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1535,13 +1542,15 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n \t * as a preferred base. Doing so can result in a larger\n \t * pack file, but the transfer will still take place.\n \t */\n+\tif (entry->type_valid)\n+\t\toe_set_size(entry, size);\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1590,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1603,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,10 +1613,11 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t\t &entry->size));\n+\t\t\t\t\t\t &size));\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n+\toe_set_size(entry, size);\n }\n \n /*\n@@ -1746,7 +1757,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (oe_size_greater_than(entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1786,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n+\tunsigned long a_size = oe_size(a);\n+\tunsigned long b_size = oe_size(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1788,9 +1801,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1877,7 +1890,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = oe_size(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1902,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = oe_size(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2022,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += oe_size(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2472,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (oe_size_less_than(entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.h b/pack-objects.h\nindex a66c37e35a..5c7e15ca92 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -69,7 +69,9 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size_;\n+\tunsigned size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n@@ -257,4 +259,43 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+static inline unsigned long oe_size(const struct object_entry *e)\n+{\n+\tif (e->size_valid) {\n+\t\treturn e->size_;\n+\t} else {\n+\t\tunsigned long size;\n+\n+\t\tsha1_object_info(e->idx.oid.hash, &size);\n+\t\treturn size;\n+\t}\n+}\n+\n+static inline int oe_size_less_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ < limit;\n+\tif (limit > maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 1;\n+\treturn oe_size(e) < limit;\n+}\n+\n+static inline int oe_size_greater_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ > limit;\n+\tif (limit <= maximum_unsigned_value_of_type(uint32_t))\n+\t\treturn 1;\n+\treturn oe_size(e) > limit;\n+}\n+\n+static inline void oe_set_size(struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\te->size_ = size;\n+\te->size_valid = e->size_ == size;\n+}\n+\n #endif\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341903","messageId":"CACsJy8BOR_YkUYONw0aofejTKuCaujRbfoea25xWMsHAT=OXuA@mail.gmail.com","threadId":"47943","inReplyTo":"87d109qh51.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 4/5] pack-objects: show some progress when counting kept objects","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:14:17Z","receivedAt":"2018-03-16T19:14:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 12, 2018 at 7:32 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Tue, Mar 06 2018, Nguyễn Thái Ngọc Duy jotted:\n>\n>> We only show progress when there are new objects to be packed. But\n>> when --keep-pack is specified on the base pack, we will exclude most\n>> of objects. This makes 'pack-objects' stay silent for a long time\n>> while the counting phase is going.\n>>\n>> Let's show some progress whenever we visit an object instead. The\n>> number of packed objects will be shown after if it's not the same as\n>> the number of visited objects.\n>>\n>> Since the meaning of this number has changed, use another word instead\n>> of \"Counting\" to hint about the change.\n>\n> Can you elaborate on how the meaning has changed? With/without this on\n> linux.git I get:\n>\n> With:\n>\n> Enumerating objects: 5901144, done.\n> Getting object details: 100% (5901145/5901145), done.\n> Delta compression using up to 8 threads.\n>\n> Without:\n>\n> Counting objects: 5901145, done.\n> Delta compression using up to 8 threads.\n>\n> So now we're seemingly off-by-one but otherwise doing the same thing?\n\nYep, it's an off-by-one bug.\n\n> As for as user feedback goes we might as well have said \"Reticulating\n> splines\", but I have some bias towards keeping the current \"Counting\n> objects...\" phrasing. We ourselves have other docs referring to it that\n> aren't changed by this patch, and there's\n> e.g. https://githubengineering.com/counting-objects/ and lots of other\n> 3rd party docs that refer to this.\n\nThis is why I changed the phrase. The counting is now a bit different.\nDocuments describing this exact phrase won't apply to the new version.\n\nThe old way counts objects that will be packed. The new way simply\ncounts objects that are visited. When you keep some packs, the number\nof objects you visit but not pack could be very high, while in normal\ncase the two numbers should be the same (e.g. you pack everything you\nvisit). I would prefer to print both values (e.g. \"counting objects:\n/\") but it's not possible with the current progress\ncode.\n-- \nDuy\n"},{"id":"341904","messageId":"20180316192745.19557-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 1/7] repack: add --keep-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:39Z","receivedAt":"2018-03-16T19:28:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We allow to keep existing packs by having companion .keep files. This\nis helpful when a pack is permanently kept. In the next patch, git-gc\njust wants to keep a pack temporarily, for one pack-objects\nrun. git-gc can use --keep-pack for this use case.\n\nA note about why the pack_keep field cannot be reused and\npack_keep_in_core has to be added. This is about the case when\n--keep-pack is specified together with either --keep-unreachable or\n--unpack-unreachable, but --honor-pack-keep is NOT specified.\n\nIn this case, we want to exclude objects from the packs specified on\ncommand line, not from ones with .keep files. If only one bit flag is\nused, we have to clear pack_keep on pack files with the .keep file.\n\nBut we can't make any assumption about unreachable objects in .keep\npacks. If \"pack_keep\" field is false for .keep packs, we could\npotentially pull lots of unreachable objects into the new pack, or\nunpack them loose. The safer approach is ignore all packs with either\n.keep file or --keep-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 +++++-\n Documentation/git-repack.txt | 9 +++++-\n builtin/pack-objects.c | 48 ++++++++++++++++++++++++++----\n builtin/repack.c | 21 +++++++++++--\n cache.h | 1 +\n t/t7700-repack.sh | 25 ++++++++++++++++\n 6 files changed, 102 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..403524652a 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]]\n+\t[--revs [--unpacked | --all]] [--keep-pack=]\n \t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n@@ -126,6 +126,13 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n+--keep-pack=::\n+\tThis flag causes an object already in the given pack to be\n+\tignored, even if it would have otherwise been\n+\tpacked. `` is the the pack file name without\n+\tleading directory (e.g. `pack-123.pack`). The option could be\n+\tspecified multiple times to keep multiple packs.\n+\n --incremental::\n \tThis flag causes an object already in a pack to be ignored\n \teven if it would have otherwise been packed.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..ce497d9d12 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=] [--keep-pack=]\n \n DESCRIPTION\n -----------\n@@ -133,6 +133,13 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--keep-pack=::\n+\tExclude the given pack from repacking. This is the equivalent\n+\tof having `.keep` file on the pack. `` is the the\n+\tpack file name without leading directory (e.g. `pack-123.pack`).\n+\tThe option could be specified multiple times to keep multiple\n+\tpacks.\n+\n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\n \tobjects older than ``. This can be used to optimize out\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..7b9fe6c89f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -28,6 +28,7 @@\n #include \"argv-array.h\"\n #include \"list.h\"\n #include \"packfile.h\"\n+#include \"dir.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -53,7 +54,7 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep;\n+static int ignore_packed_keep, ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -982,13 +983,15 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep &&\n+\tif (!ignore_packed_keep && !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n-\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\tif (p->pack_local &&\n+\t ((ignore_packed_keep && p->pack_keep) ||\n+\t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n \t/* we don't know yet; keep looking for more packs */\n@@ -2677,7 +2680,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \t\tstruct object_id oid;\n \t\tstruct object *o;\n \n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n@@ -2739,7 +2742,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \tp = (last_found != (void *)1) ? last_found : packed_git;\n \n \twhile (p) {\n-\t\tif ((!p->pack_local || p->pack_keep) &&\n+\t\tif ((!p->pack_local || p->pack_keep ||\n+\t\t\t\tp->pack_keep_in_core) &&\n \t\t\tfind_pack_entry_one(oid->hash, p)) {\n \t\t\tlast_found = p;\n \t\t\treturn 1;\n@@ -2782,7 +2786,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tstruct object_id oid;\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n \t\tif (open_pack_index(p))\n@@ -2809,6 +2813,7 @@ static int pack_options_allow_reuse(void)\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n \t !ignore_packed_keep &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n }\n@@ -2917,6 +2922,33 @@ static void get_object_list(int ac, const char **av)\n \toid_array_clear(&recent_objects);\n }\n \n+static void add_extra_kept_packs(const struct string_list *names)\n+{\n+\tstruct packed_git *p;\n+\n+\tif (!names->nr)\n+\t\treturn;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tconst char *name = basename(p->pack_name);\n+\t\tint i;\n+\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < names->nr; i++)\n+\t\t\tif (!fspathcmp(name, names->items[i].string))\n+\t\t\t\tbreak;\n+\n+\t\tif (i < names->nr) {\n+\t\t\tp->pack_keep_in_core = 1;\n+\t\t\tignore_packed_keep_in_core = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+}\n+\n static int option_parse_index_version(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -2956,6 +2988,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t N_(\"do not show progress meter\"), 0),\n@@ -3022,6 +3055,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n \t\t\t N_(\"pack compression level\")),\n \t\tOPT_SET_INT(0, \"keep-true-parents\", &grafts_replace_parents,\n@@ -3150,6 +3185,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tprogress = 2;\n \n \tprepare_packed_git();\n+\tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep) {\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..6c636e159e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n * have a corresponding .keep or .promisor file. These packs are not to\n * be kept if we are going to pack everything into one file.\n */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list)\n+static void get_non_kept_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t\tconst struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \twhile ((e = readdir(dir)) != NULL) {\n \t\tsize_t len;\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < extra_keep->nr; i++)\n+\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n+\t\t\t\tbreak;\n+\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n+\t\t\tcontinue;\n+\n \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n \t\t\tcontinue;\n \n@@ -148,7 +157,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tint ext, ret, failed;\n+\tint i, ext, ret, failed;\n \tFILE *out;\n \n \t/* variables to be filled by option parsing */\n@@ -160,6 +169,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *depth = NULL;\n \tconst char *threads = NULL;\n \tconst char *max_pack_size = NULL;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_reuse_delta = 0, no_reuse_object = 0;\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n@@ -200,6 +210,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"maximum size of each packfile\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -230,6 +242,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--keep-true-parents\");\n \tif (!pack_kept_objects)\n \t\targv_array_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list.nr; i++)\n+\t\targv_array_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t\t keep_pack_list.items[i].string);\n \targv_array_push(&cmd.args, \"--non-empty\");\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n@@ -254,7 +269,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&cmd.args, \"--write-bitmap-index\");\n \n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs);\n+\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n \t\t\tif (unpack_unreachable) {\ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..42d700f3d8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint pack_fd;\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n+\t\t pack_keep_in_core:1,\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1;\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 38247afbec..05ae0de3aa 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -196,5 +196,30 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit cat-file -t $H1\n '\n \n+test_expect_success 'repack --keep-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\ttest_commit one &&\n+\t\tgit repack -d &&\n+\t\ttest_commit two &&\n+\t\tgit repack -d &&\n+\t\ttest_commit three &&\n+\t\tgit repack -d &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 4 pack-list &&\n+\t\tKEEP1=$(head -n1 pack-list) &&\n+\t\tKEEP4=$(tail -n1 pack-list) &&\n+\t\tgit repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n+\t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\tgrep -q $KEEP1 new-counts &&\n+\t\tgrep -q $KEEP4 new-counts &&\n+\t\ttest_line_count = 3 new-counts &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n \n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341905","messageId":"20180316192745.19557-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 4/7] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:42Z","receivedAt":"2018-03-16T19:28:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\ngiant base pack to avoid this problem is also known for a long time.\n\nRecent patches add an option to do just this, but it has to be either\nconfigured or activated manually. This patch lets `git gc --auto`\nactivate this mode automatically when it thinks `repack -ad` will use\na lot of memory and start affecting the system due to swapping or\nflushing OS cache.\n\ngc --auto decides to do this based on an estimation of pack-objects\nmemory usage, which is quite accurate at least for the heap part, and\nwhether that fits in half of system memory (the assumption here is for\ndesktop environment where there are many other applications running).\n\nThis mechanism only kicks in if gc.bigBasePackThreshold is not configured.\nIf it is, it is assumed that the user already knows what they want.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 9 +++-\n builtin/gc.c | 99 +++++++++++++++++++++++++++++++++++++++-\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n t/t6500-gc.sh | 7 +++\n 7 files changed, 120 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 89f074f924..19b0d1741b 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -58,8 +58,13 @@ If the number of packs exceeds the value of `gc.autoPackLimit`,\n then existing packs (except those marked with a `.keep` file\n or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n-'git repack'. Setting `gc.autoPackLimit` to 0 disables\n-automatic consolidation of packs.\n+'git repack'.\n+If the amount of memory is estimated not enough for `git repack` to\n+run smoothly and `gc.bigPackThreshold` is not set, the largest\n+pack will also be excluded (this is the equivalent of running `git gc`\n+with `--keep-base-pack`).\n+Setting `gc.autoPackLimit` to 0 disables automatic consolidation of\n+packs.\n \n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 849f0821a9..c0f1922c24 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,10 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -40,6 +44,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static unsigned long big_pack_threshold;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -128,6 +133,7 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -167,7 +173,8 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs, unsigned long limit)\n+static struct packed_git *find_base_packs(struct string_list *packs,\n+\t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -186,6 +193,8 @@ static void find_base_packs(struct string_list *packs, unsigned long limit)\n \n \tif (base)\n \t\tstring_list_append(packs, base->pack_name);\n+\n+\treturn base;\n }\n \n static int too_many_packs(void)\n@@ -211,6 +220,79 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n+static uint64_t total_ram(void)\n+{\n+#if defined(HAVE_SYSINFO)\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tsize_t length;\n+\n+\tmib[0] = CTL_HW;\n+# if defined(HW_MEMSIZE)\n+\tmib[1] = HW_MEMSIZE;\n+# else\n+\tmib[1] = HW_PHYSMEM;\n+# endif\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo.ullTotalPhys;\n+#endif\n+\treturn 0;\n+}\n+\n+static uint64_t estimate_repack_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t os_cache, heap;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\t/*\n+\t * First we have to scan through at least one pack.\n+\t * Assume enough room in OS file cache to keep the entire pack\n+\t * or we may accidentally evict data of other processes from\n+\t * the cache.\n+\t */\n+\tos_cache = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\theap = sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\theap += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\theap += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\theap += sizeof(struct object *) * nr_objects;\n+\t/* revindex is used also */\n+\theap += sizeof(struct revindex_entry) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\theap += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\theap += max_delta_cache_size;\n+\n+\treturn os_cache + heap;\n+}\n+\n static int keep_one_pack(struct string_list_item *item, void *data)\n {\n \targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n@@ -256,6 +338,21 @@ static int need_to_gc(void)\n \n \t\tif (big_pack_threshold)\n \t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\telse {\n+\t\t\tstruct packed_git * p = find_base_packs(&keep_pack, 0);\n+\t\t\tuint64_t mem_have, mem_want;\n+\n+\t\t\tmem_have = total_ram();\n+\t\t\tmem_want = estimate_repack_memory(p);\n+\n+\t\t\t/*\n+\t\t\t * Only allow 1/2 of memory for pack-objects, leave\n+\t\t\t * the rest for the OS and other processes in the\n+\t\t\t * system.\n+\t\t\t */\n+\t\t\tif (!mem_have || mem_want < mem_have / 2)\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7b9fe6c89f..6abde6ec6d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -79,7 +79,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 68b2ad531e..a84b21986d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 4136681b47..96ca70f9cc 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -5,6 +5,13 @@ test_description='basic git gc tests\n \n . ./test-lib.sh\n \n+test_expect_success 'setup' '\n+\t# do not let the amount of physical memory affects gc\n+\t# behavior, make sure we always pack everything to one pack by\n+\t# default\n+\tgit config gc.bigPackThreshold 2g\n+'\n+\n test_expect_success 'gc empty repository' '\n \tgit gc\n '\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341906","messageId":"20180316192745.19557-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 6/7] pack-objects: show some progress when counting kept objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:44Z","receivedAt":"2018-03-16T19:28:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only show progress when there are new objects to be packed. But\nwhen --keep-pack is specified on the base pack, we will exclude most\nof objects. This makes 'pack-objects' stay silent for a long time\nwhile the counting phase is going.\n\nLet's show some progress whenever we visit an object instead. The\nnumber of packed objects will be shown after if it's not the same as\nthe number of visited objects.\n\nSince the meaning of this number has changed, use another word instead\nof \"Counting\" to hint about the change.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6abde6ec6d..f74e9117f7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -44,7 +44,7 @@ static const char *pack_usage[] = {\n static struct packing_data to_pack;\n \n static struct pack_idx_entry **written_list;\n-static uint32_t nr_result, nr_written;\n+static uint32_t nr_result, nr_written, nr_seen;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n@@ -1094,6 +1094,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n \n@@ -1109,8 +1111,6 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tcreate_object_entry(oid, type, pack_name_hash(name),\n \t\t\t exclude, name && no_try_delta(name),\n \t\t\t index_pos, found_pack, found_offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1121,6 +1121,8 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n \n@@ -1128,8 +1130,6 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, index_pos, pack, offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -3210,7 +3210,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n+\t\tprogress_state = start_progress(_(\"Enumerating objects\"), 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341907","messageId":"20180316192745.19557-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 7/7] pack-objects: display progress in get_object_details()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:45Z","receivedAt":"2018-03-16T19:28:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This code is mostly about reading object headers, which is cheap. But\nwhen the number of objects is very large (e.g. 6.5M on linux-2.6.git)\nand the system is under memory pressure, this could take some time (86\nseconds on my system).\n\nShow something during this time to let the user know pack-objects is\nstill going strong.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f74e9117f7..ac8f29dd52 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1715,6 +1715,10 @@ static void get_object_details(void)\n \tuint32_t i;\n \tstruct object_entry **sorted_by_offset;\n \n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Getting object details\"),\n+\t\t\t\t\t\tto_pack.nr_objects);\n+\n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n \tfor (i = 0; i < to_pack.nr_objects; i++)\n \t\tsorted_by_offset[i] = to_pack.objects + i;\n@@ -1725,7 +1729,9 @@ static void get_object_details(void)\n \t\tcheck_object(entry);\n \t\tif (big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n+\t\tdisplay_progress(progress_state, i + 1);\n \t}\n+\tstop_progress(&progress_state);\n \n \t/*\n \t * This must happen in a second pass, since we rely on the delta\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341908","messageId":"20180316192745.19557-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180306104158.6541-1-pclouds@gmail.com","subject":"[PATCH v3 0/7] nd/repack-keep-pack updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:38Z","receivedAt":"2018-03-16T19:28:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is not in mergeable state yet. But since there's lots of changes\nand I'm pretty sure there's still room for improvement when it comes\nto configuration and stuff, I'm posting it anyway to get the\ndiscussion continue.\n\n- --keep-pack does not imply --honor-pack-keep or --pack-kept-objects\n anymore. If you want it, you pass those in. If not, only\n --keep-pack'd packs are kept.\n\n- v2 03/05 feels too big to me so it's now broken down in a couple\n patches.\n\n- freebsd support is added and tested (which probably means OS X\n support is good too). Windows code for getting memory size remains\n untested.\n\n- platforms that do not have support for getting memory will not have\n this \"keep base pack on gc --auto\" turned on. This may be just safer\n than printing \"unrecognized platform\" which may not reach platform\n developers anyway.\n\n- gc.bigBasePackThreshold is renamed to gc.bigPackThreshold and now\n keeps all packs larger than this limit.\n\n- fix the off-by-one in \"enumerating objects\" code.\n\n- while I tend to agree to make the \"50% physical memory\"\n configurable. I'm not sure if I should add a config var for that\n (which leads to more questions like, should we allow to say \"30%\n _free_ memory\" as well?) or I should just give the memory estimation\n to a user and he/she can decide what to do with it via hooks.\n\nInterdiff\n\n-- 8< --\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 120cf6bac9..ce40112e31 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1549,12 +1549,16 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n-gc.bigBasePackThreshold::\n-\tMake `git gc --auto` only enable `--keep-base-pack` when the\n-\tbase pack's size is larger than this limit (in bytes).\n-\tDefaults to zero, which disables this check and lets\n-\t`git gc --auto` determine when to enable `--keep-base-pack`\n-\tbased on memory usage.\n+gc.bigPackThreshold::\n+\tIf non-zero, all packs larger than this limit are kept when\n+\t`git gc` is run. This is very similar to `--keep-base-pack`\n+\texcept that all packs that meet the threshold are kept, not\n+\tjust the base pack. Defaults to zero.\n++\n+Note that if the number of kept packs is more than gc.autoPackLimit,\n+this configuration variable is ignored, all packs except the base pack\n+will be repacked. After this the number of packs should go below\n+gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 35ad420d5c..19b0d1741b 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-base-pack]\n \n DESCRIPTION\n -----------\n@@ -55,15 +55,16 @@ all loose objects are combined into a single pack using\n disables automatic packing of loose objects.\n +\n If the number of packs exceeds the value of `gc.autoPackLimit`,\n-then existing packs (except those marked with a `.keep` file)\n+then existing packs (except those marked with a `.keep` file\n+or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n-'git repack'. Setting `gc.autoPackLimit` to 0 disables\n-automatic consolidation of packs.\n-+\n-If the physical amount of memory is considered not enough for `git\n-repack` to run smoothly, `--keep-base-pack` is enabled. This could be\n-overridden by setting `gc.bigBasePackThreshold` which only enables\n-`--keep-base-pack` when the base pack is larger the specified limit.\n+'git repack'.\n+If the amount of memory is estimated not enough for `git repack` to\n+run smoothly and `gc.bigPackThreshold` is not set, the largest\n+pack will also be excluded (this is the equivalent of running `git gc`\n+with `--keep-base-pack`).\n+Setting `gc.autoPackLimit` to 0 disables automatic consolidation of\n+packs.\n \n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\n@@ -84,8 +85,10 @@ overridden by setting `gc.bigBasePackThreshold` which only enables\n \tinstance running on this repository.\n \n --keep-base-pack::\n-\tAll packs except the base pack are consolidated into a single\n-\tpack. The largest pack is considered the base pack.\n+\tAll packs except the base pack and those marked with a `.keep`\n+\tfiles are consolidated into a single pack. The largest pack is\n+\tconsidered the base pack. When this option is used,\n+\t`gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\n@@ -176,10 +179,6 @@ run commands concurrently have to live with some risk of corruption (which\n seems to be low in practice) unless they turn off automatic garbage\n collection with 'git config gc.auto 0'.\n \n-Set environment variable `GIT_TRACE` in order to see the memory usage\n-estimation in `git gc --auto` that determines whether the base pack is\n-kept.\n-\n HOOKS\n -----\n \ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 1975477160..403524652a 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]]\n+\t[--revs [--unpacked | --all]] [--keep-pack=]\n \t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n@@ -126,9 +126,12 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n---keep-pack=::\n-\tIgnore the given pack. This is the equivalent of having\n-\t`.keep` file on the pack. Implies `--honor-pack-keep`.\n+--keep-pack=::\n+\tThis flag causes an object already in the given pack to be\n+\tignored, even if it would have otherwise been\n+\tpacked. `` is the the pack file name without\n+\tleading directory (e.g. `pack-123.pack`). The option could be\n+\tspecified multiple times to keep multiple packs.\n \n --incremental::\n \tThis flag causes an object already in a pack to be ignored\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 12b073e115..ce497d9d12 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=] [--keep-pack=]\n \n DESCRIPTION\n -----------\n@@ -133,9 +133,12 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n---keep-pack=::\n+--keep-pack=::\n \tExclude the given pack from repacking. This is the equivalent\n-\tof having `.keep` file on the pack. Implies `--pack-kept-objects`.\n+\tof having `.keep` file on the pack. `` is the the\n+\tpack file name without leading directory (e.g. `pack-123.pack`).\n+\tThe option could be specified multiple times to keep multiple\n+\tpacks.\n \n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ff914264a5..140c1bb7dd 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -43,7 +43,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n-static unsigned long big_base_pack_threshold;\n+static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n@@ -132,7 +132,7 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n-\tgit_config_get_ulong(\"gc.bigbasepackthreshold\", &big_base_pack_threshold);\n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config(git_default_config, NULL);\n@@ -173,18 +173,27 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static struct packed_git *find_the_base_pack(void)\n+static struct packed_git *find_base_packs(struct string_list *packs,\n+\t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n \tprepare_packed_git();\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (p->pack_local &&\n-\t\t (!base || base->pack_size < p->pack_size))\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (limit) {\n+\t\t\tif (p->pack_size >= limit)\n+\t\t\t\tstring_list_append(packs, p->pack_name);\n+\t\t} else if (!base || base->pack_size < p->pack_size) {\n \t\t\tbase = p;\n+\t\t}\n \t}\n \n+\tif (base)\n+\t\tstring_list_append(packs, base->pack_name);\n+\n \treturn base;\n }\n \n@@ -211,21 +220,24 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static inline unsigned long total_ram(void)\n+static uint64_t total_ram(void)\n {\n-\tunsigned long default_ram = 4;\n-#ifdef HAVE_SYSINFO\n+#if defined(HAVE_SYSINFO)\n \tstruct sysinfo si;\n \n \tif (!sysinfo(&si))\n \t\treturn si.totalram;\n-#elif defined(HAVE_BSD_SYSCTL) && defined(HW_MEMSIZE)\n+#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))\n \tint64_t physical_memory;\n \tint mib[2];\n \tsize_t length;\n \n \tmib[0] = CTL_HW;\n+# if defined(HW_MEMSIZE)\n \tmib[1] = HW_MEMSIZE;\n+# else\n+\tmib[1] = HW_PHYSMEM;\n+# endif\n \tlength = sizeof(int64_t);\n \tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n \t\treturn physical_memory;\n@@ -235,24 +247,18 @@ static inline unsigned long total_ram(void)\n \tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n \tif (GlobalMemoryStatusEx(&memInfo))\n \t\treturn memInfo.ullTotalPhys;\n-#else\n-\tfprintf(stderr, _(\"unrecognized platform, assuming %lu GB RAM\\n\"),\n-\t\tdefault_ram);\n #endif\n-\treturn default_ram * 1024 * 1024 * 1024;\n+\treturn 0;\n }\n \n-static int pack_objects_uses_too_much_memory(struct packed_git *pack)\n+static uint64_t estimate_repack_memory(struct packed_git *pack)\n {\n \tunsigned long nr_objects = approximate_object_count();\n-\tsize_t mem_want, mem_have, os_cache, heap;\n+\tsize_t os_cache, heap;\n \n \tif (!pack || !nr_objects)\n \t\treturn 0;\n \n-\tif (big_base_pack_threshold)\n-\t\treturn pack->pack_size >= big_base_pack_threshold;\n-\n \t/*\n \t * First we have to scan through at least one pack.\n \t * Assume enough room in OS file cache to keep the entire pack\n@@ -284,28 +290,16 @@ static int pack_objects_uses_too_much_memory(struct packed_git *pack)\n \t/* and of course pack-objects has its own delta cache */\n \theap += max_delta_cache_size;\n \n-\t/*\n-\t * Only allow 1/2 of memory for pack-objects, leave the rest\n-\t * for the OS and other processes in the system.\n-\t */\n-\tmem_have = total_ram() / 2;\n-\tmem_want = os_cache + heap;\n-\n-\ttrace_printf(\"gc mem estimation\\n\"\n-\t\t \"mem_have: %\" PRIuMAX \", mem_want: %\" PRIuMAX \", \"\n-\t\t \"heap: %\" PRIuMAX \"\\n\"\n-\t\t \"pack_size: %\" PRIuMAX \", index_size: %\" PRIuMAX \", \"\n-\t\t \"nr_objects: %\" PRIuMAX \"\\n\"\n-\t\t \"base_cache: %\" PRIuMAX \", delta_cache: %\" PRIuMAX \"\\n\",\n-\t\t (uintmax_t)mem_have, (uintmax_t)mem_want, (uintmax_t)heap,\n-\t\t (uintmax_t)pack->pack_size, (uintmax_t)pack->index_size,\n-\t\t (uintmax_t)nr_objects,\n-\t\t (uintmax_t)delta_base_cache_limit, (uintmax_t)max_delta_cache_size);\n-\n-\treturn mem_want >= mem_have;\n+\treturn os_cache + heap;\n }\n \n-static void add_repack_all_option(struct packed_git *keep_pack)\n+static int keep_one_pack(struct string_list_item *item, void *data)\n+{\n+\targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n+\treturn 0;\n+}\n+\n+static void add_repack_all_option(struct string_list *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -316,8 +310,7 @@ static void add_repack_all_option(struct packed_git *keep_pack)\n \t}\n \n \tif (keep_pack)\n-\t\targv_array_pushf(&repack, \"--keep-pack=%s\",\n-\t\t\t\t basename(keep_pack->pack_name));\n+\t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n }\n \n static void add_repack_incremental_option(void)\n@@ -341,12 +334,33 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs()) {\n-\t\tstruct packed_git *exclude = find_the_base_pack();\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n+\t\t\t\tbig_pack_threshold = 0;\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tstruct packed_git * p = find_base_packs(&keep_pack, 0);\n+\t\t\tuint64_t mem_have, mem_want;\n+\n+\t\t\tmem_have = total_ram();\n+\t\t\tmem_want = estimate_repack_memory(p);\n \n-\t\tif (!pack_objects_uses_too_much_memory(exclude))\n-\t\t\texclude = NULL;\n+\t\t\t/*\n+\t\t\t * Only allow 1/2 of memory for pack-objects, leave\n+\t\t\t * the rest for the OS and other processes in the\n+\t\t\t * system.\n+\t\t\t */\n+\t\t\tif (!mem_have || mem_want < mem_have / 2)\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t}\n \n-\t\tadd_repack_all_option(exclude);\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n \t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -558,17 +572,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tdaemonized = !daemonize();\n \t\t}\n \t} else {\n-\t\tstruct packed_git *base_pack = find_the_base_pack();\n-\t\tstruct packed_git *exclude = NULL;\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\texclude = base_pack;\n-\t\t} else if (base_pack && big_base_pack_threshold &&\n-\t\t\t base_pack->pack_size >= big_base_pack_threshold)\n-\t\t\texclude = base_pack;\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t} else if (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t}\n \n-\t\tadd_repack_all_option(exclude);\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n \tname = lock_repo_for_gc(force, &pid);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2ec911bf10..ac8f29dd52 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -54,7 +54,7 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep;\n+static int ignore_packed_keep, ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -983,13 +983,15 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep &&\n+\tif (!ignore_packed_keep && !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n-\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\tif (p->pack_local &&\n+\t ((ignore_packed_keep && p->pack_keep) ||\n+\t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n \t/* we don't know yet; keep looking for more packs */\n@@ -1092,7 +1094,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n-\tdisplay_progress(progress_state, nr_seen++);\n+\tdisplay_progress(progress_state, ++nr_seen);\n \n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n@@ -1119,7 +1121,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n-\tdisplay_progress(progress_state, nr_seen++);\n+\tdisplay_progress(progress_state, ++nr_seen);\n \n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n@@ -2684,7 +2686,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \t\tstruct object_id oid;\n \t\tstruct object *o;\n \n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n@@ -2746,7 +2748,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \tp = (last_found != (void *)1) ? last_found : packed_git;\n \n \twhile (p) {\n-\t\tif ((!p->pack_local || p->pack_keep) &&\n+\t\tif ((!p->pack_local || p->pack_keep ||\n+\t\t\t\tp->pack_keep_in_core) &&\n \t\t\tfind_pack_entry_one(oid->hash, p)) {\n \t\t\tlast_found = p;\n \t\t\treturn 1;\n@@ -2789,7 +2792,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tstruct object_id oid;\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n \t\tif (open_pack_index(p))\n@@ -2816,6 +2819,7 @@ static int pack_options_allow_reuse(void)\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n \t !ignore_packed_keep &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n }\n@@ -2939,13 +2943,14 @@ static void add_extra_kept_packs(const struct string_list *names)\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \n-\t\tfor (i = 0; i < names->nr; i++) {\n-\t\t\tif (fspathcmp(name, names->items[i].string))\n-\t\t\t\tcontinue;\n+\t\tfor (i = 0; i < names->nr; i++)\n+\t\t\tif (!fspathcmp(name, names->items[i].string))\n+\t\t\t\tbreak;\n \n-\t\t\tp->pack_keep = 1;\n-\t\t\tignore_packed_keep = 1;\n-\t\t\tbreak;\n+\t\tif (i < names->nr) {\n+\t\t\tp->pack_keep_in_core = 1;\n+\t\t\tignore_packed_keep_in_core = 1;\n+\t\t\tcontinue;\n \t\t}\n \t}\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6a1dade0e1..6c636e159e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -100,7 +100,7 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \t\tsize_t len;\n \t\tint i;\n \n-\t\tfor (i = 0;i < extra_keep->nr; i++)\n+\t\tfor (i = 0; i < extra_keep->nr; i++)\n \t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n \t\t\t\tbreak;\n \t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n@@ -227,8 +227,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))\n \t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n \n-\tif (keep_pack_list.nr && pack_kept_objects > 0)\n-\t\tdie(_(\"incompatible --keep-pack and --pack-kept-objects\"));\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps;\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..42d700f3d8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint pack_fd;\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n+\t\t pack_keep_in_core:1,\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1;\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 863fdbb0fd..96ca70f9cc 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -7,9 +7,9 @@ test_description='basic git gc tests\n \n test_expect_success 'setup' '\n \t# do not let the amount of physical memory affects gc\n-\t# behavior, make sure the pack_objects_uses_too_much_memory()\n-\t# always returns false\n-\tgit config gc.bigBasePackThreshold 2g\n+\t# behavior, make sure we always pack everything to one pack by\n+\t# default\n+\tgit config gc.bigPackThreshold 2g\n '\n \n test_expect_success 'gc empty repository' '\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 553d907d34..05ae0de3aa 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -200,16 +200,22 @@ test_expect_success 'repack --keep-pack' '\n \ttest_create_repo keep-pack &&\n \t(\n \t\tcd keep-pack &&\n-\t\tfor cmit in one two three four; do\n-\t\t\ttest_commit $cmit &&\n-\t\t\tgit repack -d\n-\t\tdone &&\n+\t\ttest_commit one &&\n+\t\tgit repack -d &&\n+\t\ttest_commit two &&\n+\t\tgit repack -d &&\n+\t\ttest_commit three &&\n+\t\tgit repack -d &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n \t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n \t\ttest_line_count = 4 pack-list &&\n-\t\tKEEP1=`head -n1 pack-list` &&\n-\t\tKEEP4=`tail -n1 pack-list` &&\n+\t\tKEEP1=$(head -n1 pack-list) &&\n+\t\tKEEP4=$(tail -n1 pack-list) &&\n \t\tgit repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n \t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\tgrep -q $KEEP1 new-counts &&\n+\t\tgrep -q $KEEP4 new-counts &&\n \t\ttest_line_count = 3 new-counts &&\n \t\tgit fsck\n \t)\n-- 8< --\n\nNguyễn Thái Ngọc Duy (7):\n repack: add --keep-pack option\n gc: add --keep-base-pack\n gc: detect base packs based on gc.bigPackThreshold config\n gc --auto: exclude base pack if not enough mem to \"repack -ad\"\n gc: handle a corner case in gc.bigPackThreshold\n pack-objects: show some progress when counting kept objects\n pack-objects: display progress in get_object_details()\n\n Documentation/config.txt | 11 ++\n Documentation/git-gc.txt | 20 +++-\n Documentation/git-pack-objects.txt | 9 +-\n Documentation/git-repack.txt | 9 +-\n builtin/gc.c | 167 +++++++++++++++++++++++++++--\n builtin/pack-objects.c | 68 +++++++++---\n builtin/repack.c | 21 +++-\n cache.h | 1 +\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 29 +++++\n t/t7700-repack.sh | 25 +++++\n 13 files changed, 339 insertions(+), 28 deletions(-)\n\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341909","messageId":"20180316192745.19557-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 5/7] gc: handle a corner case in gc.bigPackThreshold","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:43Z","receivedAt":"2018-03-16T19:28:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This config allows us to keep packs back if their size is larger\nthan a limit. But if this N >= gc.autoPackLimit, we may have a\nproblem. We are supposed to reduce the number of packs after a\nthreshold because it affects performance.\n\nWe could tell the user that they have incompatible gc.bigPackThreshold\nand gc.autoPackLimit, but it's kinda hard when 'git gc --auto' runs in\nbackground. Instead let's fall back to the next best stategy: try to\nreduce the number of packs anyway, but keep the base pack out. This\nreduces the number of packs to two and hopefully won't take up too\nmuch resources to repack (the assumption still is the base pack takes\nmost resources to handle).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 5 +++++\n builtin/gc.c | 9 +++++++--\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex c12c58813c..ce40112e31 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1554,6 +1554,11 @@ gc.bigPackThreshold::\n \t`git gc` is run. This is very similar to `--keep-base-pack`\n \texcept that all packs that meet the threshold are kept, not\n \tjust the base pack. Defaults to zero.\n++\n+Note that if the number of kept packs is more than gc.autoPackLimit,\n+this configuration variable is ignored, all packs except the base pack\n+will be repacked. After this the number of packs should go below\n+gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c0f1922c24..140c1bb7dd 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -336,9 +336,14 @@ static int need_to_gc(void)\n \tif (too_many_packs()) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold)\n+\t\tif (big_pack_threshold) {\n \t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n-\t\telse {\n+\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n+\t\t\t\tbig_pack_threshold = 0;\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t}\n+\t\t} else {\n \t\t\tstruct packed_git * p = find_base_packs(&keep_pack, 0);\n \t\t\tuint64_t mem_have, mem_want;\n \n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341910","messageId":"20180316192745.19557-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 2/7] gc: add --keep-base-pack","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:40Z","receivedAt":"2018-03-16T19:28:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This adds a new repack mode that combines everything into a secondary\npack, leaving the largest/base pack alone.\n\nThis could help reduce memory pressure. On linux-2.6.git, valgrind\nmassif reports 1.6GB heap in \"pack all\" case, and 535MB in \"pack\nall except the base pack\" case. We save roughly 1GB memory by\nexcluding the base pack.\n\nThis should also lower I/O because we don't have to rewrite a giant\npack every time (e.g. for linux-2.6.git that's a 1.4GB pack file)..\n\nPS. The use of string_list here seems overkill, but we'll need it in\nthe next patch...\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 7 +++++-\n builtin/gc.c | 47 ++++++++++++++++++++++++++++++++++++----\n t/t6500-gc.sh | 22 +++++++++++++++++++\n 3 files changed, 71 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..1717517043 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-base-pack]\n \n DESCRIPTION\n -----------\n@@ -78,6 +78,11 @@ automatic consolidation of packs.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n+--keep-base-pack::\n+\tAll packs except the base pack and those marked with a `.keep`\n+\tfiles are consolidated into a single pack. The largest pack is\n+\tconsidered the base pack.\n+\n Configuration\n -------------\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..362dd537a4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -164,6 +164,24 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n+static void find_base_packs(struct string_list *packs)\n+{\n+\tstruct packed_git *p, *base = NULL;\n+\n+\tprepare_packed_git();\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\t\tbase = p;\n+\t\t}\n+\t}\n+\n+\tif (base)\n+\t\tstring_list_append(packs, base->pack_name);\n+}\n+\n static int too_many_packs(void)\n {\n \tstruct packed_git *p;\n@@ -187,7 +205,13 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static int keep_one_pack(struct string_list_item *item, void *data)\n+{\n+\targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n+\treturn 0;\n+}\n+\n+static void add_repack_all_option(struct string_list *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -196,6 +220,9 @@ static void add_repack_all_option(void)\n \t\tif (prune_expire)\n \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n \t}\n+\n+\tif (keep_pack)\n+\t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n }\n \n static void add_repack_incremental_option(void)\n@@ -219,7 +246,7 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tadd_repack_all_option();\n+\t\tadd_repack_all_option(NULL);\n \telse if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -353,6 +380,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tconst char *name;\n \tpid_t pid;\n \tint daemonized = 0;\n+\tint keep_base_pack = -1;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -362,6 +390,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n+\t\tOPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the base pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -427,8 +457,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t */\n \t\t\tdaemonized = !daemonize();\n \t\t}\n-\t} else\n-\t\tadd_repack_all_option();\n+\t} else {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (keep_base_pack != -1) {\n+\t\t\tif (keep_base_pack)\n+\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t}\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t}\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 41b0be575d..4136681b47 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -116,6 +116,28 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_path_is_file .git/refs/heads/should-be-loose\n '\n \n+test_expect_success 'gc --keep-base-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tfor i in 10; do\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\tfor i in 10; do\n+\t\t\ttest_commit more-$i\n+\t\tdone &&\n+\t\tgit gc --keep-base-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341911","messageId":"20180316192745.19557-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v3 3/7] gc: detect base packs based on gc.bigPackThreshold config","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T19:27:41Z","receivedAt":"2018-03-16T19:28:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The --keep-base-pack option is not very convenient to use because you\nneed to tell gc to do this explicitly (and probably on just a few\nlarge repos).\n\nAdd a config key that enables this mode when packs larger than a\nlimit are found.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 6 ++++++\n Documentation/git-gc.txt | 6 ++++--\n builtin/gc.c | 26 ++++++++++++++++++++------\n 3 files changed, 30 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..c12c58813c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1549,6 +1549,12 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.bigPackThreshold::\n+\tIf non-zero, all packs larger than this limit are kept when\n+\t`git gc` is run. This is very similar to `--keep-base-pack`\n+\texcept that all packs that meet the threshold are kept, not\n+\tjust the base pack. Defaults to zero.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 1717517043..89f074f924 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -55,7 +55,8 @@ all loose objects are combined into a single pack using\n disables automatic packing of loose objects.\n +\n If the number of packs exceeds the value of `gc.autoPackLimit`,\n-then existing packs (except those marked with a `.keep` file)\n+then existing packs (except those marked with a `.keep` file\n+or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n automatic consolidation of packs.\n@@ -81,7 +82,8 @@ automatic consolidation of packs.\n --keep-base-pack::\n \tAll packs except the base pack and those marked with a `.keep`\n \tfiles are consolidated into a single pack. The largest pack is\n-\tconsidered the base pack.\n+\tconsidered the base pack. When this option is used,\n+\t`gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 362dd537a4..849f0821a9 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -39,6 +39,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_pack_threshold;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -126,6 +127,8 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -164,7 +167,7 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs)\n+static void find_base_packs(struct string_list *packs, unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -173,7 +176,10 @@ static void find_base_packs(struct string_list *packs)\n \tfor (p = packed_git; p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n-\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\tif (limit) {\n+\t\t\tif (p->pack_size >= limit)\n+\t\t\t\tstring_list_append(packs, p->pack_name);\n+\t\t} else if (!base || base->pack_size < p->pack_size) {\n \t\t\tbase = p;\n \t\t}\n \t}\n@@ -245,9 +251,15 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option(NULL);\n-\telse if (too_many_loose_objects())\n+\tif (too_many_packs()) {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (big_pack_threshold)\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -462,7 +474,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t} else if (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n-- \n2.16.2.903.gd04caf5039\n\n"},{"id":"341918","messageId":"xmqqr2oj24j1.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-9-pclouds@gmail.com","subject":"Re: [PATCH v4 08/11] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T19:40:02Z","receivedAt":"2018-03-16T20:02:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> We only cache deltas when it's smaller than a certain limit. This limit\n> defaults to 1000 but save its compressed length in a 64-bit field.\n> Shrink that field down to 16 bits, so you can only cache 65kb deltas.\n> Larger deltas must be recomputed at when the pack is written down.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\n> \t\tif (entry->delta_data && !pack_to_stdout) {\n> -\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n> -\t\t\t\t\t\t\t entry->delta_size);\n> -\t\t\tcache_lock();\n> -\t\t\tdelta_cache_size -= entry->delta_size;\n> -\t\t\tdelta_cache_size += entry->z_delta_size;\n> -\t\t\tcache_unlock();\n> +\t\t\tunsigned long size;\n> +\n> +\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n> +\t\t\tentry->z_delta_size = size;\n> +\t\t\tif (entry->z_delta_size == size) {\n\nIt is confusing to readers to write\n\n\tA = B;\n\tif (A == B) {\n\t\t/* OK, A was big enough */\n\t} else {\n\t\t/* No, B is too big to fit on A */\n\t}\n\nI actually was about to complain that you attempted an unrelated\nmicro-optimization to skip cache_lock/unlock when delta_size and\nz_delta_size are the same, and made a typo. Something like:\n\n\tsize = do_compress(...);\n\tif (size < (1 << OE_Z_DELTA_BITS)) {\n\t\tentry->z_delta_size = size;\n\t\tcache_lock();\n\t\t...\n cache_unlock();\n\t} else {\n\t\tFREE_AND_NULL(entry->delta_data);\n\t\tentry->z_delta_size = 0;\n\t}\n\nwould have saved me a few dozens of seconds of head-scratching.\n\n> +\t\t\t\tcache_lock();\n> +\t\t\t\tdelta_cache_size -= entry->delta_size;\n> +\t\t\t\tdelta_cache_size += entry->z_delta_size;\n> +\t\t\t\tcache_unlock();\n> +\t\t\t} else {\n> +\t\t\t\tFREE_AND_NULL(entry->delta_data);\n> +\t\t\t\tentry->z_delta_size = 0;\n> +\t\t\t}\n> \t\t}\n"},{"id":"341923","messageId":"xmqqmuz7242v.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-10-pclouds@gmail.com","subject":"Re: [PATCH v4 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T19:49:44Z","receivedAt":"2018-03-16T20:13:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> It's very very rare that an uncompressd object is larger than 4GB\n> (partly because Git does not handle those large files very well to\n> begin with). Let's optimize it for the common case where object size\n> is smaller than this limit.\n>\n> Shrink size field down to 32 bits [1] and one overflow bit. If the size\n> is too large, we read it back from disk.\n\nOK.\n\n> Add two compare helpers that can take advantage of the overflow\n> bit (e.g. if the file is 4GB+, chances are it's already larger than\n> core.bigFileThreshold and there's no point in comparing the actual\n> value).\n\nI had trouble reading the callers of these helpers.\n\n> +static inline int oe_size_less_than(const struct object_entry *e,\n> +\t\t\t\t unsigned long limit)\n> +{\n> +\tif (e->size_valid)\n> +\t\treturn e->size_ < limit;\n> +\tif (limit > maximum_unsigned_value_of_type(uint32_t))\n> +\t\treturn 1;\n\nWhen size_valid bit is false, that means that the size is larger\nthan 4GB. If \"limit\" is larger than 4GB, then we do not know\nanything, no? I'd understand if this \"optimization\" were\n\n\tif (limit < 4GB) {\n\t\t/*\n\t\t * we know e whose size won't fit in 4GB is larger\n\t\t * than that!\n\t\t */\n\t\treturn 0;\n\t}\n\n> +\treturn oe_size(e) < limit;\n> +}\n\nAlso, don't we want to use uintmax_t throughout the callchain? How\nwould the code in this series work when your ulong is 32-bit?\n\n> +\n> +static inline int oe_size_greater_than(const struct object_entry *e,\n> +\t\t\t\t unsigned long limit)\n> +{\n> +\tif (e->size_valid)\n> +\t\treturn e->size_ > limit;\n> +\tif (limit <= maximum_unsigned_value_of_type(uint32_t))\n> +\t\treturn 1;\n> +\treturn oe_size(e) > limit;\n> +}\n> +\n> +static inline void oe_set_size(struct object_entry *e,\n> +\t\t\t unsigned long size)\n> +{\n> +\te->size_ = size;\n> +\te->size_valid = e->size_ == size;\n> +}\n> +\n> #endif\n"},{"id":"341924","messageId":"CACsJy8C+-=z89D8V7kwPid=VG=qS_DD6yG4nhOxv=mEPN+dBow@mail.gmail.com","threadId":"47943","inReplyTo":"CACsJy8BOR_YkUYONw0aofejTKuCaujRbfoea25xWMsHAT=OXuA@mail.gmail.com","subject":"Re: [PATCH v2 4/5] pack-objects: show some progress when counting kept objects","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-16T20:13:28Z","receivedAt":"2018-03-16T20:14:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 16, 2018 at 8:14 PM, Duy Nguyen wrote:\n> On Mon, Mar 12, 2018 at 7:32 PM, Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> On Tue, Mar 06 2018, Nguyễn Thái Ngọc Duy jotted:\n>>\n>>> We only show progress when there are new objects to be packed. But\n>>> when --keep-pack is specified on the base pack, we will exclude most\n>>> of objects. This makes 'pack-objects' stay silent for a long time\n>>> while the counting phase is going.\n>>>\n>>> Let's show some progress whenever we visit an object instead. The\n>>> number of packed objects will be shown after if it's not the same as\n>>> the number of visited objects.\n>>>\n>>> Since the meaning of this number has changed, use another word instead\n>>> of \"Counting\" to hint about the change.\n>>\n>> Can you elaborate on how the meaning has changed? With/without this on\n>> linux.git I get:\n>>\n>> With:\n>>\n>> Enumerating objects: 5901144, done.\n>> Getting object details: 100% (5901145/5901145), done.\n>> Delta compression using up to 8 threads.\n>>\n>> Without:\n>>\n>> Counting objects: 5901145, done.\n>> Delta compression using up to 8 threads.\n>>\n>> So now we're seemingly off-by-one but otherwise doing the same thing?\n>\n> Yep, it's an off-by-one bug.\n>\n>> As for as user feedback goes we might as well have said \"Reticulating\n>> splines\", but I have some bias towards keeping the current \"Counting\n>> objects...\" phrasing. We ourselves have other docs referring to it that\n>> aren't changed by this patch, and there's\n>> e.g. https://githubengineering.com/counting-objects/ and lots of other\n>> 3rd party docs that refer to this.\n>\n> This is why I changed the phrase. The counting is now a bit different.\n> Documents describing this exact phrase won't apply to the new version.\n>\n> The old way counts objects that will be packed. The new way simply\n> counts objects that are visited. When you keep some packs, the number\n> of objects you visit but not pack could be very high, while in normal\n> case the two numbers should be the same (e.g. you pack everything you\n> visit). I would prefer to print both values (e.g. \"counting objects:\n> /\") but it's not possible with the current progress\n> code.\n\nOn second thought, maybe instead of introducing a new line \"getting\nobject details\" i could just rename that line to \"counting objects\"?\nThey are exactly the same, except that in the new version, this\n\"counting objects\" line could run a lot faster than the old line.\n\n\n-- \nDuy\n"},{"id":"341929","messageId":"xmqqfu4z2232.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-2-pclouds@gmail.com","subject":"Re: [PATCH v4 01/11] pack-objects: a bit of document about struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T20:32:49Z","receivedAt":"2018-03-16T20:32:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> The role of this comment block becomes more important after we shuffle\n> fields around to shrink this struct. It will be much harder to see what\n> field is related to what.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> pack-objects.h | 44 ++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 44 insertions(+)\n>\n> diff --git a/pack-objects.h b/pack-objects.h\n> index 03f1191659..85345a4af1 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -1,6 +1,50 @@\n> #ifndef PACK_OBJECTS_H\n> #define PACK_OBJECTS_H\n> \n> +/*\n> + * basic object info\n> + * -----------------\n> + * idx.oid is filled up before delta searching starts. idx.crc32 and\n> + * is only valid after the object is written out and will be used for\n\n\"and is\"?\n\n> + * generating the index. idx.offset will be both gradually set and\n> + * used in writing phase (base objects get offset first, then deltas\n> + * refer to them)\n> + *\n> + * \"size\" is the uncompressed object size. Compressed size is not\n> + * cached (ie. raw data in a pack) but available via revindex.\n\nI am having a hard time understanding what \"ie. raw data in a pack\"\nis doing in that sentence.\n\nIt is correct that compressed size is not cached; it does not even\nexist and the only way to know it is to compute it by reversing the\n.idx file (or actually uncompressing the compressed stream).\n\nPerhaps:\n\n Compressed size of the raw data for an object in a pack is not\n stored anywhere but is computed and made available when reverse\n .idx is made.\n\n> + * \"hash\" contains a path name hash which is used for sorting the\n> + * delta list and also during delta searching. Once prepare_pack()\n> + * returns it's no longer needed.\n\nHmm, that suggests an interesting optimization opportunity ;-)\n\n> + * source pack info\n> + * ----------------\n> + * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n> + * the location of the object in the source pack, with or without\n> + * header.\n\n\"with or without\", meaning...? An object in the source pack may or\nmay not have any in_pack_header, in which case in_pack_header_size\nis zero, or something? Not suggesting to rephrase (at least not\nyet), but trying to understand.\n\n> + * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n> + * may contain a delta type, while type is always the canonical type.\n> + *\n> + * deltas\n> + * ------\n> + * Delta links (delta, delta_child and delta_sibling) are created\n> + * reflect that delta graph from the source pack then updated or added\n> + * during delta searching phase when we find better deltas.\n\nIsn't anything missing after \"are created\"? Perhaps \"to\"?\n\n> + *\n> + * delta_child and delta_sibling are last needed in\n> + * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n> + * at object writing phase in case the delta is not cached.\n\nTrue. I thought child and sibling are only needed during write\norder computing, so there may be an optimization opportunity there.\n\n> + * If a delta is cached in memory and is compressed delta_data points\n\ns/compressed delta_data/compressed, delta_data/;\n\n> + * to the data and z_delta_size contains the compressed size. If it's\n> + * uncompressed [1], z_delta_size must be zero. delta_size is always\n> + * the uncompressed size and must be valid even if the delta is not\n> + * cached.\n> + *\n> + * [1] during try_delta phase we don't bother with compressing because\n> + * the delta could be quickly replaced with a better one.\n> + */\n> struct object_entry {\n> \tstruct pack_idx_entry idx;\n> \tunsigned long size;\t/* uncompressed size */\n"},{"id":"341933","messageId":"xmqqbmfn21bl.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-3-pclouds@gmail.com","subject":"Re: [PATCH v4 02/11] pack-objects: turn type and in_pack_type to bitfields","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T20:49:18Z","receivedAt":"2018-03-16T20:49:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> An extra field type_valid is added to carry the equivalent of OBJ_BAD\n> in the original \"type\" field. in_pack_type always contains a valid\n> type so we only need 3 bits for it.\n> ...\n> @@ -1570,7 +1576,7 @@ static void drop_reused_delta(struct object_entry *entry)\n> \tentry->depth = 0;\n> \n> \toi.sizep = &entry->size;\n> -\toi.typep = &entry->type;\n> +\toi.typep = &type;\n> \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n> \t\t/*\n> \t\t * We failed to get the info from this pack for some reason;\n> @@ -1578,8 +1584,10 @@ static void drop_reused_delta(struct object_entry *entry)\n> \t\t * And if that fails, the error will be recorded in entry->type\n\nThis \"entry->type\" needs updating.\n\n> \t\t * and dealt with in prepare_pack().\n> \t\t */\n> -\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n> -\t\t\t\t\t &entry->size);\n> +\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n> +\t\t\t\t\t\t &entry->size));\n> +\t} else {\n> +\t\toe_set_type(entry, type);\n> \t}\n> }\n"},{"id":"341936","messageId":"xmqq7eqb20vd.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-8-pclouds@gmail.com","subject":"Re: [PATCH v4 07/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T20:59:02Z","receivedAt":"2018-03-16T20:59:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> These delta pointers always point to elements in the objects[] array\n> in packing_data struct. We can only hold maximum 4GB of those objects\n\n4GB, as in \"number of bytes\"? Or \"We can hold 4 billion or so of\nthose objects\"?\n\n> because the array length, nr_objects, is uint32_t. We could use\n> uint32_t indexes to address these elements instead of pointers. On\n> 64-bit architecture (8 bytes per pointer) this would save 4 bytes per\n> pointer.\n>\n> Convert these delta pointers to indexes. Since we need to handle NULL\n> pointers as well, the index is shifted by one [1].\n>\n> [1] This means we can only index 2^32-2 objects even though nr_objects\n> could contain 2^32-1 objects. It should not be a problem in\n> practice because when we grow objects[], nr_alloc would probably\n> blow up long before nr_objects hits the wall.\n\n"},{"id":"341938","messageId":"xmqq370z20p5.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180316183200.31014-12-pclouds@gmail.com","subject":"Re: [PATCH v4 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T21:02:46Z","receivedAt":"2018-03-16T21:02:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Previous patches leave lots of holes and padding in this struct. This\n> patch reorders the members and shrinks the struct down to 80 bytes\n> (from 136 bytes, before any field shrinking is done) with 16 bits to\n> spare (and a couple more in in_pack_header_size when we really run out\n> of bits).\n\nNice.\n\nI am wondering if we need some conditional code for 32-bit platform.\nFor example, you have uint32_t field and do things like this:\n\n static inline int oe_size_less_than(const struct object_entry *e,\n unsigned long limit)\n {\n if (e->size_valid)\n return e->size_ < limit;\n if (limit > maximum_unsigned_value_of_type(uint32_t))\n return 1;\n return oe_size(e) < limit;\n }\n\nDo we and compilers do the right thing when your ulong is uint32_t?\n"},{"id":"341939","messageId":"871sgjent3.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-4-pclouds@gmail.com","subject":"Re: [PATCH v3 3/7] gc: detect base packs based on gc.bigPackThreshold config","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T21:02:48Z","receivedAt":"2018-03-16T21:02:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n\nAwesome, thanks for making this support N large packs.\n\n> +gc.bigPackThreshold::\n> +\tIf non-zero, all packs larger than this limit are kept when\n> +\t`git gc` is run. This is very similar to `--keep-base-pack`\n> +\texcept that all packs that meet the threshold are kept, not\n> +\tjust the base pack. Defaults to zero.\n> +\n\nWe should add:\n\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n\nSince this now supports those suffixes (yay!), see existing copy/pasting\nof that phrase in \"git help config\".\n"},{"id":"341940","messageId":"87zi37d93w.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-3-pclouds@gmail.com","subject":"Re: [PATCH v3 2/7] gc: add --keep-base-pack","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T21:05:39Z","receivedAt":"2018-03-16T21:05:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> +--keep-base-pack::\n> +\tAll packs except the base pack and those marked with a `.keep`\n> +\tfiles are consolidated into a single pack. The largest pack is\n> +\tconsidered the base pack.\n> +\n\nI wonder if all of this would be less confusing as:\n\n> +--keep-biggest-pack::\n> +\tAll packs except the largest pack and those marked with a `.keep`\n> +\tfiles are consolidated into a single pack.\n\nI.e. just skimming these docs I'd expect \"base\" to somehow be the thing\nthat we initially cloned, of course in almost all cases that *is* the\nlargest pack, but not necessarily. So rather than communicate that\nexpectation let's just say largest/biggest?\n\nMaybe I'm the only one who finds this confusing...\n"},{"id":"341941","messageId":"87y3ird8w4.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-6-pclouds@gmail.com","subject":"Re: [PATCH v3 5/7] gc: handle a corner case in gc.bigPackThreshold","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T21:10:19Z","receivedAt":"2018-03-16T21:10:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> This config allows us to keep packs back if their size is larger\n> than a limit. But if this N >= gc.autoPackLimit, we may have a\n> problem. We are supposed to reduce the number of packs after a\n> threshold because it affects performance.\n>\n> We could tell the user that they have incompatible gc.bigPackThreshold\n> and gc.autoPackLimit, but it's kinda hard when 'git gc --auto' runs in\n> background. Instead let's fall back to the next best stategy: try to\n> reduce the number of packs anyway, but keep the base pack out. This\n> reduces the number of packs to two and hopefully won't take up too\n> much resources to repack (the assumption still is the base pack takes\n> most resources to handle).\n\nI think this strategy makes perfect sense.\n\nThose with say a 1GB \"base\" pack might set this setting at to 500MB or\nsomething large like that, then it's realistically never going to happen\nthat you're going to then have a collision between gc.bigPackThreshold\nand gc.autoPackLimit, even if your checkout is many years old *maybe*\nyou've accumulated 5-10 of those 500MB packs for any sane repo.\n\nBut this also allows for setting this value really low, e.g. 50MB or\nsomething to place a very low upper bound on how much memory GC takes on\na regular basis, but of course you'll need to repack that set of 50MB's\neventually.\n\nGreat!\n"},{"id":"341942","messageId":"87woybd8pl.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-5-pclouds@gmail.com","subject":"Re: [PATCH v3 4/7] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T21:14:14Z","receivedAt":"2018-03-16T21:14:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n\n\n> +\t\t\tstruct packed_git * p = find_base_packs(&keep_pack, 0);\n\nStyle nit: space after \"*\".\n"},{"id":"341946","messageId":"87vadvd87n.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-3-pclouds@gmail.com","subject":"Re: [PATCH v3 2/7] gc: add --keep-base-pack","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-16T21:25:00Z","receivedAt":"2018-03-16T21:25:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> \tstruct option builtin_gc_options[] = {\n> \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n> @@ -362,6 +390,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n> \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n> \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n> +\t\tOPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n> +\t\t\t N_(\"repack all other packs except the base pack\")),\n> \t\tOPT_END()\n> \t};\n\nThere's an easy to solve merge conflict here between the current master\n& this. Pushed out a solution (for my own use) at\nhttps://github.com/avar/git/ gc-auto-keep-base-pack. Interdiff with\nyours:\n\n @@ -112,9 +112,9 @@\n struct option builtin_gc_options[] = {\n OPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n @@\n - OPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n - OPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n - OPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n + OPT_BOOL_F(0, \"force\", &force,\n + N_(\"force running gc even if there may be another gc running\"),\n + PARSE_OPT_NOCOMPLETE),\n + OPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n + N_(\"repack all other packs except the base pack\")),\n OPT_END()\n"},{"id":"341951","messageId":"xmqqpo43zoul.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"xmqqmuz7242v.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-16T21:34:42Z","receivedAt":"2018-03-16T21:34:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Also, don't we want to use uintmax_t throughout the callchain? How\n> would the code in this series work when your ulong is 32-bit?\n\nMy own answer to this question is \"no conversion to uintmax_t, at\nleast not in this series.\" As long as the original code uses\n\"unsigned long\", this series also should, I think.\n\n"},{"id":"342015","messageId":"CACsJy8CVQBgs4s8VyQ2PoVx3kVv+n76ctwjCe_2YkLGiWd2r2A@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqfu4z2232.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 01/11] pack-objects: a bit of document about struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T11:59:38Z","receivedAt":"2018-03-17T12:00:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 16, 2018 at 9:32 PM, Junio C Hamano wrote:\n>> +/*\n>> + * basic object info\n>> + * -----------------\n>> + * idx.oid is filled up before delta searching starts. idx.crc32 and\n>> + * is only valid after the object is written out and will be used for\n>\n> \"and is\"?\n\nThere was another field that I thought was only valid after blah blah.\nBut it was wrong and I forgot to delete this \"and\" after deleting that\nfield.\n\n>> + * \"hash\" contains a path name hash which is used for sorting the\n>> + * delta list and also during delta searching. Once prepare_pack()\n>> + * returns it's no longer needed.\n>\n> Hmm, that suggests an interesting optimization opportunity ;-)\n\nHeh.. it does not reduce peak memory consumption though which is why\nI'm less interested in freeing it after prepare_pack().\n\n>> + * source pack info\n>> + * ----------------\n>> + * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n>> + * the location of the object in the source pack, with or without\n>> + * header.\n>\n> \"with or without\", meaning...? An object in the source pack may or\n> may not have any in_pack_header, in which case in_pack_header_size\n> is zero, or something? Not suggesting to rephrase (at least not\n> yet), but trying to understand.\n\nThe location with the header (i.e. true beginning an object in a pack)\nor without/after the header so you are at the zlib stream, ready to\ninflate or reuse. I'll rephrase this a bit.\n\n>> + *\n>> + * delta_child and delta_sibling are last needed in\n>> + * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n>> + * at object writing phase in case the delta is not cached.\n>\n> True. I thought child and sibling are only needed during write\n> order computing, so there may be an optimization opportunity there.\n\nSee. I wrote all this for a reason. Somebody looking for low hang\nfruit can always find some ;-)\n-- \nDuy\n"},{"id":"342016","messageId":"CACsJy8BwhE5fFOOmJo7Eb68Q52+pJQMHWeBFRNLfZGgcm1QWBg@mail.gmail.com","threadId":"47943","inReplyTo":"xmqq370z20p5.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T12:07:25Z","receivedAt":"2018-03-17T12:08:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 16, 2018 at 10:02 PM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> Previous patches leave lots of holes and padding in this struct. This\n>> patch reorders the members and shrinks the struct down to 80 bytes\n>> (from 136 bytes, before any field shrinking is done) with 16 bits to\n>> spare (and a couple more in in_pack_header_size when we really run out\n>> of bits).\n>\n> Nice.\n>\n> I am wondering if we need some conditional code for 32-bit platform.\n> For example, you have uint32_t field and do things like this:\n>\n> static inline int oe_size_less_than(const struct object_entry *e,\n> unsigned long limit)\n> {\n> if (e->size_valid)\n> return e->size_ < limit;\n> if (limit > maximum_unsigned_value_of_type(uint32_t))\n> return 1;\n> return oe_size(e) < limit;\n> }\n>\n> Do we and compilers do the right thing when your ulong is uint32_t?\n\nAnother good point. My 32-bit build does complain\n\nIn file included from builtin/pack-objects.c:20:0:\n./pack-objects.h: In function ?oe_size_less_than?:\n./pack-objects.h:282:12: error: comparison is always false due to\nlimited range of data type [-Werror=type-limits]\n if (limit > maximum_unsigned_value_of_type(uint32_t))\n ^\n-- \nDuy\n"},{"id":"342019","messageId":"20180317141033.21545-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-1-pclouds@gmail.com","subject":"[PATCH v5 00/11] nd/pack-objects-pack-struct updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:22Z","receivedAt":"2018-03-17T14:10:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v5 changes are small enough that the interdiff is pretty self\nexplanatory (there's also a couple commit msg updates).\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c388d87c3e..fb2aba80bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1611,7 +1611,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n-\t\t * And if that fails, the error will be recorded in entry->type\n+\t\t * And if that fails, the error will be recorded in oe_type(entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n@@ -1968,7 +1968,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n-\tif (delta_size >= maximum_unsigned_value_of_type(uint32_t))\n+\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n \t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n@@ -2125,8 +2125,8 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t\tunsigned long size;\n \n \t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n-\t\t\tentry->z_delta_size = size;\n-\t\t\tif (entry->z_delta_size == size) {\n+\t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n+\t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n \t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0fa0c83294..8979289f5f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -27,14 +27,15 @@ enum dfs_state {\n *\n * basic object info\n * -----------------\n- * idx.oid is filled up before delta searching starts. idx.crc32 and\n- * is only valid after the object is written out and will be used for\n+ * idx.oid is filled up before delta searching starts. idx.crc32 is\n+ * only valid after the object is written out and will be used for\n * generating the index. idx.offset will be both gradually set and\n * used in writing phase (base objects get offset first, then deltas\n * refer to them)\n *\n- * \"size\" is the uncompressed object size. Compressed size is not\n- * cached (ie. raw data in a pack) but available via revindex.\n+ * \"size\" is the uncompressed object size. Compressed size of the raw\n+ * data for an object in a pack is not stored anywhere but is computed\n+ * and made available when reverse .idx is made.\n *\n * \"hash\" contains a path name hash which is used for sorting the\n * delta list and also during delta searching. Once prepare_pack()\n@@ -42,16 +43,16 @@ enum dfs_state {\n *\n * source pack info\n * ----------------\n- * The (in_pack, in_pack_offset, in_pack_header_size) tuple contains\n- * the location of the object in the source pack, with or without\n- * header.\n+ * The (in_pack, in_pack_offset) tuple contains the location of the\n+ * object in the source pack. in_pack_header_size allows quickly\n+ * skipping the header and going straight to the zlib stream.\n *\n * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n * may contain a delta type, while type is always the canonical type.\n *\n * deltas\n * ------\n- * Delta links (delta, delta_child and delta_sibling) are created\n+ * Delta links (delta, delta_child and delta_sibling) are created to\n * reflect that delta graph from the source pack then updated or added\n * during delta searching phase when we find better deltas.\n *\n@@ -59,7 +60,7 @@ enum dfs_state {\n * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n * at object writing phase in case the delta is not cached.\n *\n- * If a delta is cached in memory and is compressed delta_data points\n+ * If a delta is cached in memory and is compressed, delta_data points\n * to the data and z_delta_size contains the compressed size. If it's\n * uncompressed [1], z_delta_size must be zero. delta_size is always\n * the uncompressed size and must be valid even if the delta is not\n@@ -274,12 +275,19 @@ static inline unsigned long oe_size(const struct object_entry *e)\n \t}\n }\n \n+static inline int contains_in_32bits(unsigned long limit)\n+{\n+\tuint32_t truncated_limit = (uint32_t)limit;\n+\n+\treturn limit == truncated_limit;\n+}\n+\n static inline int oe_size_less_than(const struct object_entry *e,\n \t\t\t\t unsigned long limit)\n {\n \tif (e->size_valid)\n \t\treturn e->size_ < limit;\n-\tif (limit > maximum_unsigned_value_of_type(uint32_t))\n+\tif (contains_in_32bits(limit))\n \t\treturn 1;\n \treturn oe_size(e) < limit;\n }\n@@ -289,8 +297,8 @@ static inline int oe_size_greater_than(const struct object_entry *e,\n {\n \tif (e->size_valid)\n \t\treturn e->size_ > limit;\n-\tif (limit <= maximum_unsigned_value_of_type(uint32_t))\n-\t\treturn 1;\n+\tif (contains_in_32bits(limit))\n+\t\treturn 0;\n \treturn oe_size(e) > limit;\n }\n \n@@ -314,7 +322,7 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n \t\t\t\t unsigned long size)\n {\n \te->delta_size_ = size;\n-\te->delta_size_valid =e->delta_size_ == size;\n+\te->delta_size_valid = e->delta_size_ == size;\n \tif (!e->delta_size_valid && size != oe_size(e))\n \t\tdie(\"BUG: this can only happen in check_object() \"\n \t\t \"where delta size is the same as entry size\");\n\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342020","messageId":"20180317141033.21545-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 01/11] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:23Z","receivedAt":"2018-03-17T14:10:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..de91edd264 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,51 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 is\n+ * only valid after the object is written out and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size of the raw\n+ * data for an object in a pack is not stored anywhere but is computed\n+ * and made available when reverse .idx is made.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset) tuple contains the location of the\n+ * object in the source pack. in_pack_header_size allows quickly\n+ * skipping the header and going straight to the zlib stream.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created to\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342021","messageId":"20180317141033.21545-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 03/11] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:25Z","receivedAt":"2018-03-17T14:11:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 28 +++++++++++++++++-----------\n 2 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 647c01ea34..83f8154865 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 5f568b609c..4c6b73a4d6 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -73,19 +88,10 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n+\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342022","messageId":"20180317141033.21545-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:26Z","receivedAt":"2018-03-17T14:11:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Because of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 5 ++---\n 5 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 83f8154865..829c80ffcc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth >= (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS));\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 4c6b73a4d6..a4d8d29c04 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -89,9 +90,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\tint depth;\n-\n+\tunsigned depth:OE_DEPTH_BITS;\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342023","messageId":"20180317141033.21545-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 07/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:29Z","receivedAt":"2018-03-17T14:11:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These delta pointers always point to elements in the objects[] array\nin packing_data struct. We can only hold maximum 4G of those objects\nbecause the array size in nr_objects is uint32_t. We could use\nuint32_t indexes to address these elements instead of pointers. On\n64-bit architecture (8 bytes per pointer) this would save 4 bytes per\npointer.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 67 ++++++++++++++++++++++--\n 2 files changed, 124 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex eaf78fa41a..379bd1ab92 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -489,7 +495,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -541,12 +547,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -608,34 +614,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -646,7 +652,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -661,8 +667,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -672,11 +678,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1498,10 +1504,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1572,17 +1578,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1622,7 +1630,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1647,7 +1655,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1670,7 +1678,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1685,7 +1693,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1870,7 +1878,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1946,7 +1954,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1975,7 +1983,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1984,13 +1992,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2059,7 +2067,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2109,7 +2117,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2117,7 +2125,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2438,7 +2446,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 933f71a86b..0b831c8f12 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -72,11 +72,11 @@ struct object_entry {\n \tunsigned long size;\t/* uncompressed size */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -200,4 +200,61 @@ static inline void oe_set_in_pack(struct object_entry *e,\n \n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342024","messageId":"20180317141033.21545-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 08/11] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:30Z","receivedAt":"2018-03-17T14:11:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 16 bits, so you can only cache 65kb deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 22 ++++++++++++++++------\n pack-objects.h | 3 ++-\n 3 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 379bd1ab92..71ca1ba2ce 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2105,12 +2105,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n+\t\t\t\tentry->z_delta_size = size;\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3089,6 +3096,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS));\n+\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n+\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n+\t\t 1 << OE_Z_DELTA_BITS);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0b831c8f12..63222a76b0 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n+#define OE_Z_DELTA_BITS\t\t16\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -79,7 +80,7 @@ struct object_entry {\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342025","messageId":"20180317141033.21545-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:31Z","receivedAt":"2018-03-17T14:11:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressd object is larger than 4GB\n(partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size\nis smaller than this limit.\n\nShrink size field down to 32 bits [1] and one overflow bit. If the size\nis too large, we read it back from disk.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nA small note about the conditional oe_set_size() in\ncheck_object(). Technically if we don't get a valid type, it's not\nwrong if we set uninitialized value \"size\" (we don't pre-initialize\nthis and sha1_object_info will not assign anything when it fails to\nget the info).\n\nThis how changes the writing code path slightly which emits different\nerror messages (either way we die). One of our tests in t5530 depends\non this specific error message. Let's just keep the test as-is and\nplay safe by not assigning random value. That might trigger valgrind\nanyway.\n\n[1] it's actually already 32 bits on Windows\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 49 ++++++++++++++++++++++++++---------------\n pack-objects.h | 50 +++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 80 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 71ca1ba2ce..887e12c556 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -274,7 +274,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +384,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = oe_size(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +407,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1412,6 +1413,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1431,13 +1434,14 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n+\t\toe_set_size(entry, size);\n \n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n@@ -1505,7 +1509,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = oe_size(entry);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1513,14 +1517,17 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = get_size_from_delta(p, &w_curs,\n+\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n-\t\t\tif (entry->size == 0)\n+\t\t\toe_set_size(entry, size);\n+\t\t\tif (oe_size_less_than(entry, 1))\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1535,13 +1542,15 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n \t * as a preferred base. Doing so can result in a larger\n \t * pack file, but the transfer will still take place.\n \t */\n+\tif (entry->type_valid)\n+\t\toe_set_size(entry, size);\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1590,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1603,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,10 +1613,11 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t\t &entry->size));\n+\t\t\t\t\t\t &size));\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n+\toe_set_size(entry, size);\n }\n \n /*\n@@ -1746,7 +1757,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (oe_size_greater_than(entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1786,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n+\tunsigned long a_size = oe_size(a);\n+\tunsigned long b_size = oe_size(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1788,9 +1801,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1877,7 +1890,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = oe_size(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1902,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = oe_size(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2022,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += oe_size(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2472,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (oe_size_less_than(entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 63222a76b0..9a4ed7fdbe 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -70,7 +70,9 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size_;\n+\tunsigned size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n@@ -258,4 +260,50 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+static inline unsigned long oe_size(const struct object_entry *e)\n+{\n+\tif (e->size_valid) {\n+\t\treturn e->size_;\n+\t} else {\n+\t\tunsigned long size;\n+\n+\t\tsha1_object_info(e->idx.oid.hash, &size);\n+\t\treturn size;\n+\t}\n+}\n+\n+static inline int contains_in_32bits(unsigned long limit)\n+{\n+\tuint32_t truncated_limit = (uint32_t)limit;\n+\n+\treturn limit == truncated_limit;\n+}\n+\n+static inline int oe_size_less_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ < limit;\n+\tif (contains_in_32bits(limit))\n+\t\treturn 1;\n+\treturn oe_size(e) < limit;\n+}\n+\n+static inline int oe_size_greater_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ > limit;\n+\tif (contains_in_32bits(limit))\n+\t\treturn 0;\n+\treturn oe_size(e) > limit;\n+}\n+\n+static inline void oe_set_size(struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\te->size_ = size;\n+\te->size_valid = e->size_ == size;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342026","messageId":"20180317141033.21545-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:33Z","receivedAt":"2018-03-17T14:11:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Previous patches leave lots of holes and padding in this struct. This\npatch reorders the members and shrinks the struct down to 80 bytes\n(from 136 bytes, before any field shrinking is done) with 16 bits to\nspare (and a couple more in in_pack_header_size when we really run out\nof bits).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 17 +++++++++--------\n 1 file changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 2507b157d5..8979289f5f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -71,35 +71,36 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\t/* object uncompressed size _if_ size_valid is true */\n-\tuint32_t size_;\n-\tunsigned size_valid:1;\n-\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n+\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tunsigned size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n-\tunsigned type_valid:1;\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n \tunsigned no_try_delta:1;\n+\tunsigned char in_pack_header_size;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n+\n+\t/* size: 80, bit_padding: 16 bits */\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342027","messageId":"20180317141033.21545-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 10/11] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:32Z","receivedAt":"2018-03-17T14:11:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n31 bits with one overflow bit.\n\nIf we find an existing delta larger than 2GB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 24 ++++++++++++++----------\n pack-objects.h | 23 ++++++++++++++++++++++-\n 2 files changed, 36 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 887e12c556..fb2aba80bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,10 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -140,7 +142,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -291,14 +293,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1509,7 +1511,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = oe_size(entry);\n+\t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1895,7 +1897,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -1966,10 +1968,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n+\t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -1984,7 +1988,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -1997,7 +2001,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2120,11 +2124,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 9a4ed7fdbe..2507b157d5 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,7 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_DELTA_SIZE_BITS\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -81,7 +82,8 @@ struct object_entry {\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tuint32_t delta_size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -306,4 +308,23 @@ static inline void oe_set_size(struct object_entry *e,\n \te->size_valid = e->size_ == size;\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid = e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(e))\n+\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342028","messageId":"20180317141033.21545-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 02/11] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:24Z","receivedAt":"2018-03-17T14:11:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"An extra field type_valid is added to carry the equivalent of OBJ_BAD\nin the original \"type\" field. in_pack_type always contains a valid\ntype so we only need 3 bits for it.\n\nA note about accepting OBJ_NONE as \"valid\" type. The function\nread_object_list_from_stdin() can pass this value [1] and it\neventually calls create_object_entry() where current code skip setting\n\"type\" field if the incoming type is zero. This does not have any bad\nside effects because \"type\" field should be memset()'d anyway.\n\nBut since we also need to set type_valid now, skipping oe_set_type()\nleaves type_valid zero/false, which will make oe_type() return\nOBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in\nprepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger\n\n fatal: unable to get type of object ...\n\nAccepting OBJ_NONE [2] does sound wrong, but this is how it is has\nbeen for a very long time and I haven't time to dig in further.\n\n[1] See 5c49c11686 (pack-objects: better check_object() performances -\n 2007-04-16)\n\n[2] 21666f1aae (convert object type handling from a string to a number\n - 2007-02-26)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 60 ++++++++++++++++++++++++------------------\n cache.h | 2 ++\n object.h | 1 -\n pack-bitmap-write.c | 6 ++---\n pack-objects.h | 20 ++++++++++++--\n 5 files changed, 58 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..647c01ea34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1066,8 +1067,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1428,9 +1433,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1484,7 +1489,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1493,7 +1498,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1516,7 +1521,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,16 +1576,18 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n-\t\t * And if that fails, the error will be recorded in entry->type\n+\t\t * And if that fails, the error will be recorded in oe_type(entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n+\t} else {\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1747,10 +1755,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1826,7 +1836,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2432,11 +2442,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2545,7 +2555,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..fd11f08940 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \n \t\tentry->in_pack_pos = i;\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -98,7 +98,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex de91edd264..5f568b609c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -59,8 +59,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n+\tunsigned type_:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -123,4 +124,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342029","messageId":"20180317141033.21545-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 05/11] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:27Z","receivedAt":"2018-03-17T14:11:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 16 +++++++++++++++-\n 5 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 829c80ffcc..727d200770 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -879,7 +879,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fd11f08940..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex a4d8d29c04..b832ee2b5e 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -79,7 +79,6 @@ struct object_entry {\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -99,6 +98,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -144,4 +145,17 @@ static inline void oe_set_type(struct object_entry *e,\n \te->type_ = (unsigned)type;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342030","messageId":"20180317141033.21545-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v5 06/11] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-17T14:10:28Z","receivedAt":"2018-03-17T14:11:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index instead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 16k. For now if you hit\n16k pack files limit, pack-objects will simply fail [1].\n\n[1] The escape hatch is .keep file to limit the non-kept pack files\n below 16k limit. Then you can go for another pack-objects run to\n combine another 16k pack files. Repeat until you're satisfied.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++++\n builtin/pack-objects.c | 40 +++++++++++++++++----------\n cache.h | 1 +\n pack-objects.h | 44 +++++++++++++++++++++++++++++-\n 4 files changed, 79 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 3503c9e3e6..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 727d200770..eaf78fa41a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -1025,7 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1048,11 +1050,16 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index)\n+\t\toe_add_pack(&to_pack, *found_pack);\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1074,7 +1081,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1406,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1542,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1587,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1857,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -3191,6 +3200,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\t/* make sure IN_PACK(0) return NULL */\n+\toe_add_pack(&to_pack, NULL);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b832ee2b5e..933f71a86b 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t14\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -18,6 +19,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 is\n@@ -65,7 +70,7 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -100,6 +105,8 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -158,4 +165,39 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline unsigned int oe_add_pack(struct packing_data *pack,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t \"Please add .keep files to exclude\\n\"\n+\t\t \"some pack files and keep the number \"\n+\t\t \"of non-kept files below %d.\"),\n+\t\t 1 << OE_IN_PACK_BITS);\n+\tif (p) {\n+\t\tif (p->index > 0)\n+\t\t\tdie(\"BUG: this packed is already indexed\");\n+\t\tp->index = pack->in_pack_count;\n+\t}\n+\tpack->in_pack[pack->in_pack_count] = p;\n+\treturn pack->in_pack_count++;\n+}\n+\n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack[e->in_pack_idx];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (p->index <= 0)\n+\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t \"instead of having non-positive index\");\n+\te->in_pack_idx = p->index;\n+\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342044","messageId":"87po42cwql.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"Re: [PATCH v5 00/11] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T19:45:06Z","receivedAt":"2018-03-17T19:47:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 17 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> v5 changes are small enough that the interdiff is pretty self\n> explanatory (there's also a couple commit msg updates).\n\nI've been testing this and it's definitely an improvement. I think it\nwould be good to get some mention in the commit messages themselves of\nthe incremental improvement, to that end I wrote this:\n\n $ cat /tmp/howmuch-mem.sh\n #!/bin/sh\n cd /tmp &&\n (\n for i in {1..3}\n do\n /usr/bin/time -f MaxRSS:%M ~/g/git/git --git-dir=/tmp/linux.git --exec-path=/home/avar/g/git repack -A -d 2>&1\n done | grep MaxRSS: | sort -n | head -n 1 | tr '\\n' '\\t' &&\n git git-commit-summary &&\n echo\n ) | tee -a /tmp/git-memory.log\n\nI.e. we repack linux.git (I'd already repacked it once) and do three\nruns, and take the lowest RSS size. This yields (I rebased your series\non top of git@github.com:git/git.git master and pushed it to\ngit@github.com:avar/git.git pack-objects-reduce-memory-footprint), via:\n\n git rebase --exec='make -j8 CFLAGS=\"-O3\" && /tmp/howmuch-mem.sh' -i\n\nThat gave me, kb on the first column:\n\n MaxRSS:3746648 f23a196dd9 (\"pack-objects: a bit of document about struct object_entry\", 2018-03-01)\n MaxRSS:3700696 953b6473d7 (\"pack-objects: turn type and in_pack_type to bitfields\", 2018-03-01)\n MaxRSS:3700404 6cbe573539 (\"pack-objects: use bitfield for object_entry::dfs_state\", 2018-03-01)\n MaxRSS:3654044 0b93ebcae9 (\"pack-objects: use bitfield for object_entry::depth\", 2018-03-01)\n MaxRSS:3654040 67a4d48773 (\"pack-objects: move in_pack_pos out of struct object_entry\", 2018-03-01) [X]\n MaxRSS:3654104 e77319c65a (\"pack-objects: move in_pack out of struct object_entry\", 2018-03-01) [X]\n MaxRSS:3608096 a72cfcfea3 (\"pack-objects: refer to delta objects by index instead of pointer\", 2018-03-01)\n MaxRSS:3562212 76eaa779eb (\"pack-objects: shrink z_delta_size field in struct object_entry\", 2018-03-05)\n MaxRSS:3515164 42e28dd4b3 (\"pack-objects: shrink size field in struct object_entry\", 2018-03-05)\n MaxRSS:3469440 26eba3ded4 (\"pack-objects: shrink delta_size field in struct object_entry\", 2018-03-05)\n MaxRSS:3423704 c6493de964 (\"pack-objects.h: reorder members to shrink struct object_entry\", 2018-03-12)\n\nI.e. on git.git we end up with just over a a 8.5% reduction, and\ninterestingly have a slight increase over a past commit in one change,\nand one that just makes 4kb of difference (marked via [X] above).\n\nAlso, your v0 says it overall saves 260MB of memory. According to this\nit's 320MB. You did note some reductions in subsequent patches, but it's\nworth calling that out explicitly.\n\nI have a bigger in-house repo that looks like this with this change:\n\n MaxRSS:4753120 f23a196dd9 (\"pack-objects: a bit of document about struct object_entry\", 2018-03-01)\n MaxRSS:4699084 953b6473d7 (\"pack-objects: turn type and in_pack_type to bitfields\", 2018-03-01)\n MaxRSS:4699028 6cbe573539 (\"pack-objects: use bitfield for object_entry::dfs_state\", 2018-03-01)\n MaxRSS:4645452 0b93ebcae9 (\"pack-objects: use bitfield for object_entry::depth\", 2018-03-01)\n MaxRSS:4645288 67a4d48773 (\"pack-objects: move in_pack_pos out of struct object_entry\", 2018-03-01)\n MaxRSS:4645548 e77319c65a (\"pack-objects: move in_pack out of struct object_entry\", 2018-03-01)\n MaxRSS:4591484 a72cfcfea3 (\"pack-objects: refer to delta objects by index instead of pointer\", 2018-03-01)\n MaxRSS:4537980 76eaa779eb (\"pack-objects: shrink z_delta_size field in struct object_entry\", 2018-03-05)\n MaxRSS:4484148 42e28dd4b3 (\"pack-objects: shrink size field in struct object_entry\", 2018-03-05)\n MaxRSS:4430404 26eba3ded4 (\"pack-objects: shrink delta_size field in struct object_entry\", 2018-03-05)\n MaxRSS:4376148 c6493de964 (\"pack-objects.h: reorder members to shrink struct object_entry\", 2018-03-12)\n\nI.e. a tad more than a 7.9% reduction in memory use.\n\nThis series also doesn't make a difference to the total runtime (which\nis good, just wanted to make sure). On linux.git on my box best out of\nthree is 1:15.74 before and 1:14.93 after, which is within the margin of\nrandom error.\n"},{"id":"342045","messageId":"87o9jmcwmf.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"87po42cwql.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 00/11] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T19:47:36Z","receivedAt":"2018-03-17T19:47:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 17 2018, Ævar Arnfjörð Bjarmason jotted:\n\n> [...]I.e. on git.git we end up with just over a a 8.5% reduction, and[...]\n\nUrgh, sorry, this should say \"on linux.git...\". None of these numbers\ncame from testing git.git.\n"},{"id":"342046","messageId":"87muz6cwcu.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-12-pclouds@gmail.com","subject":"Re: [PATCH v5 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T19:53:21Z","receivedAt":"2018-03-17T19:53:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 17 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> Previous patches leave lots of holes and padding in this struct. This\n> patch reorders the members and shrinks the struct down to 80 bytes\n> (from 136 bytes, before any field shrinking is done) with 16 bits to\n> spare (and a couple more in in_pack_header_size when we really run out\n> of bits).\n\nGiven what I mentioned in 87po42cwql.fsf@evledraar.gmail.com just now I\nthink we should add this to the commit message.\n\n This is the last in a series of memory reduction patches (see\n \"pack-objects: a bit of document about struct object_entry\" for the\n first one).\n\n Overall they've reduced repack memory size on linux.git from 3.747G\n to 3.424G, or by around 320M, a decrease of 8.5%. The runtime of\n repack has stayed the same throughout this series. Ævar's testing on\n a big monorepo he has access to (bigger than linux.git) has shown a\n 7.9% reduction, so the overall expected improvement should be\n somewhere around 8%.\n\n See 87po42cwql.fsf@evledraar.gmail.com on-list\n (https://public-inbox.org/git/87po42cwql.fsf@evledraar.gmail.com/)\n for more detailed numbers and a test script used to produce the\n numbers cited above.\n\nThanks again for working on this.\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> pack-objects.h | 17 +++++++++--------\n> 1 file changed, 9 insertions(+), 8 deletions(-)\n>\n> diff --git a/pack-objects.h b/pack-objects.h\n> index 2507b157d5..8979289f5f 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -71,35 +71,36 @@ enum dfs_state {\n> */\n> struct object_entry {\n> \tstruct pack_idx_entry idx;\n> -\t/* object uncompressed size _if_ size_valid is true */\n> -\tuint32_t size_;\n> -\tunsigned size_valid:1;\n> -\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n> +\tvoid *delta_data;\t/* cached delta (uncompressed) */\n> \toff_t in_pack_offset;\n> +\tuint32_t hash;\t\t\t/* name hint hash */\n> +\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n> \tuint32_t delta_idx;\t/* delta base object */\n> \tuint32_t delta_child_idx; /* deltified objects who bases me */\n> \tuint32_t delta_sibling_idx; /* other deltified objects who\n> \t\t\t\t * uses the same base as me\n> \t\t\t\t */\n> -\tvoid *delta_data;\t/* cached delta (uncompressed) */\n> \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n> \tuint32_t delta_size_valid:1;\n> +\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n> +\tunsigned size_valid:1;\n> \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n> +\tunsigned type_valid:1;\n> \tunsigned type_:TYPE_BITS;\n> \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n> -\tunsigned type_valid:1;\n> -\tuint32_t hash;\t\t\t/* name hint hash */\n> -\tunsigned char in_pack_header_size;\n> \tunsigned preferred_base:1; /*\n> \t\t\t\t * we do not pack this, but is available\n> \t\t\t\t * to be used as the base object to delta\n> \t\t\t\t * objects against.\n> \t\t\t\t */\n> \tunsigned no_try_delta:1;\n> +\tunsigned char in_pack_header_size;\n> \tunsigned tagged:1; /* near the very tip of refs */\n> \tunsigned filled:1; /* assigned write-order */\n> \tunsigned dfs_state:OE_DFS_STATE_BITS;\n> \tunsigned depth:OE_DEPTH_BITS;\n> +\n> +\t/* size: 80, bit_padding: 16 bits */\n> };\n>\n> struct packing_data {\n"},{"id":"342047","messageId":"87lgeqcw6k.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-10-pclouds@gmail.com","subject":"Re: [PATCH v5 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T19:57:07Z","receivedAt":"2018-03-17T19:57:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 17 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> It's very very rare that an uncompressd object is larger than 4GB\n\ns/uncompressd/uncompressed/\n"},{"id":"342051","messageId":"87h8pecs1u.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-5-pclouds@gmail.com","subject":"Re: [PATCH v5 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T21:26:21Z","receivedAt":"2018-03-17T21:26:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 17 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> Because of struct packing from now on we can only handle max depth\n> 4095\n> [...]\n> +\tif (depth >= (1 << OE_DEPTH_BITS))\n> +\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n> +\t\t depth, (1 << OE_DEPTH_BITS));\n> +\n\nThis has a off-by-one error:\n\n $ git repack --depth=4096\n fatal: delta chain depth 4096 is greater than maximum limit 4096\n\nPer the check we should be feeding `(1 << OE_DEPTH_BITS) - 1` to die().\n"},{"id":"342057","messageId":"87fu4ycq7s.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180228092722.GA25627@ash","subject":"Why does pack-objects use so much memory on incremental packing?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-17T22:05:59Z","receivedAt":"2018-03-17T22:06:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 28 2018, Duy Nguyen jotted:\n\n> linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n> all apps nearly unusuable (granted the problem is partly Linux I/O\n> scheduler too). So I wonder if we can reduce pack-objects memory\n> footprint a bit.\n>\n> This demonstration patch (probably breaks some tests) would reduce the\n> size of struct object_entry from from 136 down to 112 bytes on\n> x86-64. There are 6483999 of these objects, so the saving is 17% or\n> 148 MB.\n\nSplitting this off into its own thread. Aside from the improvements in\nyour repack memory reduction (20180317141033.21545-1-pclouds@gmail.com)\nand gc config (20180316192745.19557-1-pclouds@gmail.com) series's I'm\nwondering why repack takes so much memory to incrementally repack new\nstuff when you leave out the base pack.\n\nRepacking git.git takes around 290MB of memory on my system, but I'd\nthink that this would make it take a mere few megabytes, since all I'm\nasking it to do is pack up the few loose objects that got added and keep\nthe base pack:\n\n (\n rm -rf /tmp/git &&\n git clone git@github.com:git/git.git /tmp/git &&\n cd /tmp/git &&\n touch $(ls .git/objects/pack/*pack | sed 's/\\.pack$/.keep/') &&\n for i in {1..10}\n do\n touch $i &&\n git add $i &&\n git commit -m$i\n done &&\n /usr/bin/time -f %M git repack -A -d\n )\n\nBut no, it takes around 230MB. But thinking about it a bit further:\n\n * This builds on top of existing history, so that needs to be\n read/consulted\n\n * We might be reusing (if not directly, skipping re-comuting) deltas\n from the existing pack.\n\nBut I get the same result if after cloning I make an orphan branch, and\npass all the \"do this as cheaply as possible\" branches I can find down\nto git-repack:\n\n (\n rm -rf /tmp/git &&\n git clone git@github.com:git/git.git /tmp/git &&\n cd /tmp/git &&\n touch $(ls .git/objects/pack/*pack | sed 's/\\.pack$/.keep/') &&\n git checkout --orphan new &&\n git reset --hard &&\n for i in {1..10}\n do\n touch $i &&\n git add $i &&\n git commit -m$i\n done &&\n git tag -d $(git tag -l) &&\n /usr/bin/time -f %M git repack -A -d -f -F --window=1 --depth=1\n )\n\nBut the memory use barely changes, my first example used 227924 kb, but\nthis one uses 226788.\n\nOf course nobody's going to clone a huge repo and then right away create\nan --orphan branch, but is there an inherent reason for why this\ncouldn't be taking as much memory as if the repo was cloned with\n--depth=1?\n\nI.e. when I have a *.keep on an existing pack we would have some\nlow-memory mode to copy the trees/blobs needed for the current commit\nover to the new pack, and use that as the basis for packing everything\ngoing forward.\n\nJeff: Is this something ref islands[1] could be (ab)used to do, or have\nI misunderstood that concept?\n\n1. https://public-inbox.org/git/20130626051117.GB26755@sigill.intra.peff.net/\n https://public-inbox.org/git/20160304153359.GA16300@sigill.intra.peff.net/\n https://public-inbox.org/git/20160809174528.2ydgkhd7ayclat3t@sigill.intra.peff.net/\n"},{"id":"342075","messageId":"xmqq8taqx949.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180317141033.21545-10-pclouds@gmail.com","subject":"Re: [PATCH v5 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-18T05:09:42Z","receivedAt":"2018-03-18T05:09:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> +static inline int contains_in_32bits(unsigned long limit)\n> +{\n\nThis name somehow does not sound right.\n\nIf the verb \"contain\" must be used, the way to phrase what this\nfunction does with the verb is to say \"limit can be contained in a\n32-bit int\", so \"contains\" is probably where the funniness comes\nfrom.\n\n\"fits in 32bits\" is OK, I think.\n\n> +\tuint32_t truncated_limit = (uint32_t)limit;\n> +\n> +\treturn limit == truncated_limit;\n> +}\n\nI am guessing that a compiler that is clever enough will make this\nfunction a no-op on a 32-bit arch and that is why it is a static\ninline function?\n\n> +static inline int oe_size_less_than(const struct object_entry *e,\n> +\t\t\t\t unsigned long limit)\n> +{\n> +\tif (e->size_valid)\n> +\t\treturn e->size_ < limit;\n\ne->size_ is the true size so we can compare it to see if it is smaller\nthan limit.\n\n> +\tif (contains_in_32bits(limit))\n> +\t\treturn 1;\n\nIf limit is small enough, and because e->size_valid means e->size_\ndoes not fit in 32-bit, we know size is larger than limit.\nShouldn't we be returning 0 that means \"no, the size is not less\nthan limit\" from here?\n\n> +\treturn oe_size(e) < limit;\n> +}\n> +\n> +static inline int oe_size_greater_than(const struct object_entry *e,\n> +\t\t\t\t unsigned long limit)\n> +{\n> +\tif (e->size_valid)\n> +\t\treturn e->size_ > limit;\n\ne->size_ is the true size so we compare and return if it is larger\nthan limit.\n\n> +\tif (contains_in_32bits(limit))\n> +\t\treturn 0;\n\nNow e->size_ is larger than what would fit within 32-bit. If limit\nfits within 32-bit, then size must be larger than limit. Again,\nshouldn't we be returning 1 that means \"yes, the size is greater\nthan limit\" from here?\n\n> +\treturn oe_size(e) > limit;\n> +}\n> +\n> +static inline void oe_set_size(struct object_entry *e,\n> +\t\t\t unsigned long size)\n> +{\n> +\te->size_ = size;\n> +\te->size_valid = e->size_ == size;\n> +}\n> +\n> #endif\n"},{"id":"342079","messageId":"CACsJy8AXSCw1b0FpsUv85kZ_HYhHy9OHeDZGGFFg9pOno8GfNw@mail.gmail.com","threadId":"47943","inReplyTo":"xmqq8taqx949.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T08:23:05Z","receivedAt":"2018-03-18T08:23:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Mar 18, 2018 at 6:09 AM, Junio C Hamano wrote:\n>> + uint32_t truncated_limit = (uint32_t)limit;\n>> +\n>> + return limit == truncated_limit;\n>> +}\n>\n> I am guessing that a compiler that is clever enough will make this\n> function a no-op on a 32-bit arch and that is why it is a static\n> inline function?\n\nIt's a separate function because I don't want to duplicate this ==\nlogic twice. Even if the compiler does not optimize this, it's still\nmuch cheaper than oe_sze() which involves disk access.\n\n>> +static inline int oe_size_less_than(const struct object_entry *e,\n>> + unsigned long limit)\n>> +{\n>> + if (e->size_valid)\n>> + return e->size_ < limit;\n>\n> e->size_ is the true size so we can compare it to see if it is smaller\n> than limit.\n>\n>> + if (contains_in_32bits(limit))\n>> + return 1;\n>\n> If limit is small enough, and because e->size_valid means e->size_\n> does not fit in 32-bit, we know size is larger than limit.\n> Shouldn't we be returning 0 that means \"no, the size is not less\n> than limit\" from here?\n\nArgh!!! This logic keeps messing with my brain.\n-- \nDuy\n"},{"id":"342081","messageId":"CACsJy8B8YucY5cJEL9ku2yUhcD4Js2XAjsmHP1B0sGa3yixGhw@mail.gmail.com","threadId":"47943","inReplyTo":"87fu4ycq7s.fsf@evledraar.gmail.com","subject":"Re: Why does pack-objects use so much memory on incremental packing?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T08:37:39Z","receivedAt":"2018-03-18T08:38:15Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 17, 2018 at 11:05 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Wed, Feb 28 2018, Duy Nguyen jotted:\n>\n>> linux-2.6.git current has 6483999 objects. \"git gc\" on my poor laptop\n>> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making\n>> all apps nearly unusuable (granted the problem is partly Linux I/O\n>> scheduler too). So I wonder if we can reduce pack-objects memory\n>> footprint a bit.\n>>\n>> This demonstration patch (probably breaks some tests) would reduce the\n>> size of struct object_entry from from 136 down to 112 bytes on\n>> x86-64. There are 6483999 of these objects, so the saving is 17% or\n>> 148 MB.\n>\n> Splitting this off into its own thread. Aside from the improvements in\n> your repack memory reduction (20180317141033.21545-1-pclouds@gmail.com)\n> and gc config (20180316192745.19557-1-pclouds@gmail.com) series's I'm\n> wondering why repack takes so much memory to incrementally repack new\n> stuff when you leave out the base pack.\n>\n> Repacking git.git takes around 290MB of memory on my system, but I'd\n> think that this would make it take a mere few megabytes, since all I'm\n> asking it to do is pack up the few loose objects that got added and keep\n> the base pack:\n>\n> ...\n>\n\nI left some clue in the new estimate_repack_memory() function in my gc\nseries that could help you find this out. I haven't really tested this\ncase but my guess is the two cache pools we have will likely be filled\nup close to full anyway and hit delta_base_cache_limit and\nmax_delta_cache_size limits. When these are really full on default\nconfiguration, they'll take roughly ~300mb.\n\nThe second is, I think we still go through all objects to mark which\none is included in the new pack, which one not (and probably which one\ncan be delta base candidates). Try calling alloc_report() function at\nthe end of repack to see exactly how much memory is locked in there.\nThis we could perhaps improve for incremental repack by avoiding\nrunning rev-list.\n-- \nDuy\n"},{"id":"342083","messageId":"CACsJy8D8ZrzyNE=JAGEnENhtGnhHvB4vL2B-tex=Z2=D2Mx4ZQ@mail.gmail.com","threadId":"47943","inReplyTo":"87muz6cwcu.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 11/11] pack-objects.h: reorder members to shrink struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T08:49:22Z","receivedAt":"2018-03-18T08:49:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 17, 2018 at 8:53 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Sat, Mar 17 2018, Nguyễn Thái Ngọc Duy jotted:\n>\n>> Previous patches leave lots of holes and padding in this struct. This\n>> patch reorders the members and shrinks the struct down to 80 bytes\n>> (from 136 bytes, before any field shrinking is done) with 16 bits to\n>> spare (and a couple more in in_pack_header_size when we really run out\n>> of bits).\n>\n> Given what I mentioned in 87po42cwql.fsf@evledraar.gmail.com just now I\n> think we should add this to the commit message.\n>\n> This is the last in a series of memory reduction patches (see\n> \"pack-objects: a bit of document about struct object_entry\" for the\n> first one).\n>\n> Overall they've reduced repack memory size on linux.git from 3.747G\n> to 3.424G, or by around 320M, a decrease of 8.5%. The runtime of\n> repack has stayed the same throughout this series. Ævar's testing on\n> a big monorepo he has access to (bigger than linux.git) has shown a\n> 7.9% reduction, so the overall expected improvement should be\n> somewhere around 8%.\n>\n> See 87po42cwql.fsf@evledraar.gmail.com on-list\n> (https://public-inbox.org/git/87po42cwql.fsf@evledraar.gmail.com/)\n> for more detailed numbers and a test script used to produce the\n> numbers cited above.\n\nYeah.\n\nI probably should add something that was on my mind but never written\nout. These shrinking and packing definitely slow down access to these\nstruct members (more instructions to read or write). However, since\npack-objects is mostly IO-bound, and when it's CPU-bound, I think the\nhot path is either inflating objects or generating deltas, these\nslowdowns do not matter (and smaller cache footprint helps too)\n-- \nDuy\n"},{"id":"342093","messageId":"20180318142526.9378-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180317141033.21545-1-pclouds@gmail.com","subject":"[PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:15Z","receivedAt":"2018-03-18T14:25:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v6 fixes the one optimization that I just couldn't get right, fixes\ntwo off-by-one error messages and a couple commit message update\n(biggest change is in 11/11 to record some numbers from AEvar)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fb2aba80bf..4406af640f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3112,10 +3112,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n-\t\t depth, (1 << OE_DEPTH_BITS));\n+\t\t depth, (1 << OE_DEPTH_BITS) - 1);\n \tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n \t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n-\t\t 1 << OE_Z_DELTA_BITS);\n+\t\t (1 << OE_Z_DELTA_BITS) - 1);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 55358da9f3..af40211105 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -275,7 +275,7 @@ static inline unsigned long oe_size(const struct object_entry *e)\n \t}\n }\n \n-static inline int contains_in_32bits(unsigned long limit)\n+static inline int oe_fits_in_32bits(unsigned long limit)\n {\n \tuint32_t truncated_limit = (uint32_t)limit;\n \n@@ -287,8 +287,8 @@ static inline int oe_size_less_than(const struct object_entry *e,\n {\n \tif (e->size_valid)\n \t\treturn e->size_ < limit;\n-\tif (contains_in_32bits(limit))\n-\t\treturn 1;\n+\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\t\treturn 0;\n \treturn oe_size(e) < limit;\n }\n \n@@ -297,8 +297,8 @@ static inline int oe_size_greater_than(const struct object_entry *e,\n {\n \tif (e->size_valid)\n \t\treturn e->size_ > limit;\n-\tif (contains_in_32bits(limit))\n-\t\treturn 0;\n+\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\t\treturn 1;\n \treturn oe_size(e) > limit;\n }\n \n@@ -307,6 +307,14 @@ static inline void oe_set_size(struct object_entry *e,\n {\n \te->size_ = size;\n \te->size_valid = e->size_ == size;\n+\n+\tif (!e->size_valid) {\n+\t\tunsigned long real_size;\n+\n+\t\tif (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n+\t\t size != real_size)\n+\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n+\t}\n }\n \n static inline unsigned long oe_delta_size(struct packing_data *pack,\n\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342094","messageId":"20180318142526.9378-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 01/11] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:16Z","receivedAt":"2018-03-18T14:25:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..c0a1f61aac 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,51 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 is\n+ * only valid after the object is written out and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size of the raw\n+ * data for an object in a pack is not stored anywhere but is computed\n+ * and made available when reverse .idx is made.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset) tuple contains the location of the\n+ * object in the source pack. in_pack_header_size allows quickly\n+ * skipping the header and going straight to the zlib stream.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created to\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342095","messageId":"20180318142526.9378-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 02/11] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:17Z","receivedAt":"2018-03-18T14:25:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"An extra field type_valid is added to carry the equivalent of OBJ_BAD\nin the original \"type\" field. in_pack_type always contains a valid\ntype so we only need 3 bits for it.\n\nA note about accepting OBJ_NONE as \"valid\" type. The function\nread_object_list_from_stdin() can pass this value [1] and it\neventually calls create_object_entry() where current code skip setting\n\"type\" field if the incoming type is zero. This does not have any bad\nside effects because \"type\" field should be memset()'d anyway.\n\nBut since we also need to set type_valid now, skipping oe_set_type()\nleaves type_valid zero/false, which will make oe_type() return\nOBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in\nprepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger\n\n fatal: unable to get type of object ...\n\nAccepting OBJ_NONE [2] does sound wrong, but this is how it is has\nbeen for a very long time and I haven't time to dig in further.\n\n[1] See 5c49c11686 (pack-objects: better check_object() performances -\n 2007-04-16)\n\n[2] 21666f1aae (convert object type handling from a string to a number\n - 2007-02-26)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 60 ++++++++++++++++++++++++------------------\n cache.h | 2 ++\n object.h | 1 -\n pack-bitmap-write.c | 6 ++---\n pack-objects.h | 20 ++++++++++++--\n 5 files changed, 58 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..647c01ea34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1066,8 +1067,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1428,9 +1433,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1484,7 +1489,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1493,7 +1498,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1516,7 +1521,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,16 +1576,18 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n-\t\t * And if that fails, the error will be recorded in entry->type\n+\t\t * And if that fails, the error will be recorded in oe_type(entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n+\t} else {\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1747,10 +1755,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1826,7 +1836,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2432,11 +2442,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2545,7 +2555,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..fd11f08940 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \n \t\tentry->in_pack_pos = i;\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -98,7 +98,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex c0a1f61aac..b883d7aa10 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -59,8 +59,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n+\tunsigned type_:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -123,4 +124,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342096","messageId":"20180318142526.9378-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 03/11] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:18Z","receivedAt":"2018-03-18T14:25:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 28 +++++++++++++++++-----------\n 2 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 647c01ea34..83f8154865 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b883d7aa10..8507e1b869 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -73,19 +88,10 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n+\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342097","messageId":"20180318142526.9378-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 04/11] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:19Z","receivedAt":"2018-03-18T14:25:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Because of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 5 ++---\n 5 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 83f8154865..205e1f646c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth >= (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS) - 1);\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 8507e1b869..59407aae3c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -89,9 +90,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\tint depth;\n-\n+\tunsigned depth:OE_DEPTH_BITS;\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342098","messageId":"20180318142526.9378-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 05/11] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:20Z","receivedAt":"2018-03-18T14:26:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 16 +++++++++++++++-\n 5 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 205e1f646c..e1244918a5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -879,7 +879,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fd11f08940..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 59407aae3c..4a11653657 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -79,7 +79,6 @@ struct object_entry {\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -99,6 +98,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -144,4 +145,17 @@ static inline void oe_set_type(struct object_entry *e,\n \te->type_ = (unsigned)type;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342099","messageId":"20180318142526.9378-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 08/11] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:23Z","receivedAt":"2018-03-18T14:26:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 16 bits, so you can only cache 65kb deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 22 ++++++++++++++++------\n pack-objects.h | 3 ++-\n 3 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b39234f7fb..372afe48c4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2105,12 +2105,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n+\t\t\t\tentry->z_delta_size = size;\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3089,6 +3096,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS) - 1);\n+\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n+\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n+\t\t (1 << OE_Z_DELTA_BITS) - 1);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 594a213554..c12219385a 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n+#define OE_Z_DELTA_BITS\t\t16\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -79,7 +80,7 @@ struct object_entry {\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342100","messageId":"20180318142526.9378-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 06/11] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:21Z","receivedAt":"2018-03-18T14:26:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index instead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 16k. For now if you hit\n16k pack files limit, pack-objects will simply fail [1].\n\n[1] The escape hatch is .keep file to limit the non-kept pack files\n below 16k limit. Then you can go for another pack-objects run to\n combine another 16k pack files. Repeat until you're satisfied.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++++\n builtin/pack-objects.c | 40 +++++++++++++++++----------\n cache.h | 1 +\n pack-objects.h | 44 +++++++++++++++++++++++++++++-\n 4 files changed, 79 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 3503c9e3e6..b8d936ccf5 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,6 +269,15 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n+LIMITATIONS\n+-----------\n+\n+This command could only handle 16384 existing pack files at a time.\n+If you have more than this, you need to exclude some pack files with\n+\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n+in one, then remove these .keep files and run pack-objects one more\n+time.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e1244918a5..9792d31e46 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -1025,7 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\treturn want;\n+\t\t\tgoto done;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1048,11 +1050,16 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\treturn want;\n+\t\t\t\tgoto done;\n \t\t}\n \t}\n \n-\treturn 1;\n+\twant = 1;\n+done:\n+\tif (want && *found_pack && !(*found_pack)->index)\n+\t\toe_add_pack(&to_pack, *found_pack);\n+\n+\treturn want;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1074,7 +1081,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1406,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1542,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1587,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1857,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -3191,6 +3200,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\t/* make sure IN_PACK(0) return NULL */\n+\toe_add_pack(&to_pack, NULL);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 4a11653657..bf905c3f9b 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t14\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -18,6 +19,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 is\n@@ -65,7 +70,7 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -100,6 +105,8 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\tint in_pack_count;\n+\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -158,4 +165,39 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline unsigned int oe_add_pack(struct packing_data *pack,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n+\t\tdie(_(\"too many packs to handle in one go. \"\n+\t\t \"Please add .keep files to exclude\\n\"\n+\t\t \"some pack files and keep the number \"\n+\t\t \"of non-kept files below %d.\"),\n+\t\t 1 << OE_IN_PACK_BITS);\n+\tif (p) {\n+\t\tif (p->index > 0)\n+\t\t\tdie(\"BUG: this packed is already indexed\");\n+\t\tp->index = pack->in_pack_count;\n+\t}\n+\tpack->in_pack[pack->in_pack_count] = p;\n+\treturn pack->in_pack_count++;\n+}\n+\n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack[e->in_pack_idx];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (p->index <= 0)\n+\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t \"instead of having non-positive index\");\n+\te->in_pack_idx = p->index;\n+\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342101","messageId":"20180318142526.9378-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 07/11] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:22Z","receivedAt":"2018-03-18T14:26:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These delta pointers always point to elements in the objects[] array\nin packing_data struct. We can only hold maximum 4G of those objects\nbecause the array size in nr_objects is uint32_t. We could use\nuint32_t indexes to address these elements instead of pointers. On\n64-bit architecture (8 bytes per pointer) this would save 4 bytes per\npointer.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 67 ++++++++++++++++++++++--\n 2 files changed, 124 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9792d31e46..b39234f7fb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -489,7 +495,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -541,12 +547,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -608,34 +614,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -646,7 +652,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -661,8 +667,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -672,11 +678,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1498,10 +1504,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1572,17 +1578,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1622,7 +1630,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1647,7 +1655,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1670,7 +1678,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1685,7 +1693,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1870,7 +1878,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1946,7 +1954,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1975,7 +1983,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1984,13 +1992,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2059,7 +2067,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2109,7 +2117,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2117,7 +2125,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2438,7 +2446,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex bf905c3f9b..594a213554 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -72,11 +72,11 @@ struct object_entry {\n \tunsigned long size;\t/* uncompressed size */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -200,4 +200,61 @@ static inline void oe_set_in_pack(struct object_entry *e,\n \n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342102","messageId":"20180318142526.9378-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 11/11] pack-objects: reorder members to shrink struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:26Z","receivedAt":"2018-03-18T14:26:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Previous patches leave lots of holes and padding in this struct. This\npatch reorders the members and shrinks the struct down to 80 bytes\n(from 136 bytes, before any field shrinking is done) with 16 bits to\nspare (and a couple more in in_pack_header_size when we really run out\nof bits).\n\nThis is the last in a series of memory reduction patches (see\n\"pack-objects: a bit of document about struct object_entry\" for the\nfirst one).\n\nOverall they've reduced repack memory size on linux-2.6.git from\n3.747G to 3.424G, or by around 320M, a decrease of 8.5%. The runtime\nof repack has stayed the same throughout this series. Ævar's testing\non a big monorepo he has access to (bigger than linux-2.6.git) has\nshown a 7.9% reduction, so the overall expected improvement should be\nsomewhere around 8%.\n\nSee 87po42cwql.fsf@evledraar.gmail.com on-list\n(https://public-inbox.org/git/87po42cwql.fsf@evledraar.gmail.com/) for\nmore detailed numbers and a test script used to produce the numbers\ncited above.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 17 +++++++++--------\n 1 file changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex cbd5cf61ca..af40211105 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -71,35 +71,36 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\t/* object uncompressed size _if_ size_valid is true */\n-\tuint32_t size_;\n-\tunsigned size_valid:1;\n-\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n+\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tunsigned size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n-\tunsigned type_valid:1;\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n \tunsigned no_try_delta:1;\n+\tunsigned char in_pack_header_size;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n \tunsigned depth:OE_DEPTH_BITS;\n+\n+\t/* size: 80, bit_padding: 16 bits */\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342103","messageId":"20180318142526.9378-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:24Z","receivedAt":"2018-03-18T14:26:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressedd object is larger than 4GB\n(partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size\nis smaller than this limit.\n\nShrink size field down to 32 bits [1] and one overflow bit. If the size\nis too large, we read it back from disk.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nA small note about the conditional oe_set_size() in\ncheck_object(). Technically if we don't get a valid type, it's not\nwrong if we set uninitialized value \"size\" (we don't pre-initialize\nthis and sha1_object_info will not assign anything when it fails to\nget the info).\n\nThis how changes the writing code path slightly which emits different\nerror messages (either way we die). One of our tests in t5530 depends\non this specific error message. Let's just keep the test as-is and\nplay safe by not assigning random value. That might trigger valgrind\nanyway.\n\n[1] it's actually already 32 bits on Windows\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 49 ++++++++++++++++++++++-------------\n pack-objects.h | 58 +++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 88 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 372afe48c4..89ed4b5125 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -274,7 +274,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +384,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = oe_size(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +407,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1412,6 +1413,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1431,13 +1434,14 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n+\t\toe_set_size(entry, size);\n \n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n@@ -1505,7 +1509,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = oe_size(entry);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1513,14 +1517,17 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = get_size_from_delta(p, &w_curs,\n+\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n-\t\t\tif (entry->size == 0)\n+\t\t\toe_set_size(entry, size);\n+\t\t\tif (oe_size_less_than(entry, 1))\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1535,13 +1542,15 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n \t * as a preferred base. Doing so can result in a larger\n \t * pack file, but the transfer will still take place.\n \t */\n+\tif (entry->type_valid)\n+\t\toe_set_size(entry, size);\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1590,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1603,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,10 +1613,11 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t\t &entry->size));\n+\t\t\t\t\t\t &size));\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n+\toe_set_size(entry, size);\n }\n \n /*\n@@ -1746,7 +1757,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (oe_size_greater_than(entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1786,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n+\tunsigned long a_size = oe_size(a);\n+\tunsigned long b_size = oe_size(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1788,9 +1801,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1877,7 +1890,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = oe_size(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1902,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = oe_size(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2022,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += oe_size(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2472,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (oe_size_less_than(entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.h b/pack-objects.h\nindex c12219385a..0beedbc637 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -70,7 +70,9 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size_;\n+\tunsigned size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n@@ -258,4 +260,58 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+static inline unsigned long oe_size(const struct object_entry *e)\n+{\n+\tif (e->size_valid) {\n+\t\treturn e->size_;\n+\t} else {\n+\t\tunsigned long size;\n+\n+\t\tsha1_object_info(e->idx.oid.hash, &size);\n+\t\treturn size;\n+\t}\n+}\n+\n+static inline int oe_fits_in_32bits(unsigned long limit)\n+{\n+\tuint32_t truncated_limit = (uint32_t)limit;\n+\n+\treturn limit == truncated_limit;\n+}\n+\n+static inline int oe_size_less_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ < limit;\n+\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\t\treturn 0;\n+\treturn oe_size(e) < limit;\n+}\n+\n+static inline int oe_size_greater_than(const struct object_entry *e,\n+\t\t\t\t unsigned long limit)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_ > limit;\n+\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\t\treturn 1;\n+\treturn oe_size(e) > limit;\n+}\n+\n+static inline void oe_set_size(struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\te->size_ = size;\n+\te->size_valid = e->size_ == size;\n+\n+\tif (!e->size_valid) {\n+\t\tunsigned long real_size;\n+\n+\t\tif (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n+\t\t size != real_size)\n+\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n+\t}\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342104","messageId":"20180318142526.9378-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v6 10/11] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-18T14:25:25Z","receivedAt":"2018-03-18T14:26:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n31 bits with one overflow bit.\n\nIf we find an existing delta larger than 2GB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 24 ++++++++++++++----------\n pack-objects.h | 23 ++++++++++++++++++++++-\n 2 files changed, 36 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 89ed4b5125..4406af640f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,10 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -140,7 +142,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -291,14 +293,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1509,7 +1511,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = oe_size(entry);\n+\t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1895,7 +1897,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -1966,10 +1968,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n+\t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -1984,7 +1988,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -1997,7 +2001,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2120,11 +2124,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0beedbc637..cbd5cf61ca 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,7 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_DELTA_SIZE_BITS\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -81,7 +82,8 @@ struct object_entry {\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tuint32_t delta_size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -314,4 +316,23 @@ static inline void oe_set_size(struct object_entry *e,\n \t}\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid = e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(e))\n+\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.17.0.rc0.347.gf9cf61673a\n\n"},{"id":"342105","messageId":"878tapcucc.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-10-pclouds@gmail.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-18T14:49:07Z","receivedAt":"2018-03-18T14:49:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Mar 18 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> It's very very rare that an uncompressedd object is larger than 4GB\n\nSo this went from a typo of \"uncompressd\" in v5 to \"uncompressedd\",\nneeds one less \"d\": \"uncompressed\".\n"},{"id":"342106","messageId":"877eq9cu96.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-18T14:51:01Z","receivedAt":"2018-03-18T14:51:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Mar 18 2018, Nguyễn Thái Ngọc Duy jotted:\n\n> v6 fixes the one optimization that I just couldn't get right, fixes\n> two off-by-one error messages and a couple commit message update\n> (biggest change is in 11/11 to record some numbers from AEvar)\n\nThanks, aside from the minor typo I just noted in\nhttps://public-inbox.org/git/878tapcucc.fsf@evledraar.gmail.com/ (which\nI trust Junio can fix up) this all looks good to me.\n"},{"id":"342196","messageId":"xmqqzi34vy01.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180318142526.9378-10-pclouds@gmail.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-19T16:19:42Z","receivedAt":"2018-03-19T16:19:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> +static inline int oe_fits_in_32bits(unsigned long limit)\n> +{\n> +\tuint32_t truncated_limit = (uint32_t)limit;\n> +\n> +\treturn limit == truncated_limit;\n> +}\n\nI do not think it is worth a reroll (there only are a few\ncallsites), but the above has nothing to do with \"oe\" fitting\nanything (it is about \"limit\"). Do you mind if I did this instead?\n\n\tstatic inline int fits_in_32bits(unsigned long size)\n\n... or other suggestions, perhaps?\n\n"},{"id":"342197","messageId":"CACsJy8CFZhUfCuzkKTYg9-_qrz4OtkZD3B3-4_P0B8KTgNR4cQ@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqzi34vy01.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-19T16:23:45Z","receivedAt":"2018-03-19T16:24:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 19, 2018 at 5:19 PM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> +static inline int oe_fits_in_32bits(unsigned long limit)\n>> +{\n>> + uint32_t truncated_limit = (uint32_t)limit;\n>> +\n>> + return limit == truncated_limit;\n>> +}\n>\n> I do not think it is worth a reroll (there only are a few\n> callsites), but the above has nothing to do with \"oe\" fitting\n> anything (it is about \"limit\"). Do you mind if I did this instead?\n>\n> static inline int fits_in_32bits(unsigned long size)\n>\n> ... or other suggestions, perhaps?\n>\n\nI just tried to not pollute the general namespace too much. That works\nfor me too.\n-- \nDuy\n"},{"id":"342199","messageId":"xmqqsh8wvwwn.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180318142526.9378-10-pclouds@gmail.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-19T16:43:20Z","receivedAt":"2018-03-19T16:43:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> +static inline void oe_set_size(struct object_entry *e,\n> +\t\t\t unsigned long size)\n> +{\n> +\te->size_ = size;\n> +\te->size_valid = e->size_ == size;\n\nA quite similar comment as my earlier one applies here. I wonder if\nthis is easier to read?\n\t\n\te->size_valid = fits_in_32bits(size);\n\tif (e->size_valid)\n\t\te->size_ = size;\n\nStepping back a bit in a different tangent, \n\n - fits_in_32bits() is a good public name if the helper is about\n seeing if the given quantity fits in 32bit uint,\n\n - but that carves it in stone that our e->size_ *will* be 32bit\n forever, which is not good.\n\nSo, it may be a good idea to call it size_cacheable_in_oe(size) or\nsomething to ask \"I have this 'size'; is it small enough to fit in\nthe field in the oe, i.e. allow us to cache it, as opposed to having\nto go back to the object every time?\" Of course, this would declare\nthat the helper can only be used for that particular field, but that\nis sort of the point of such a change, to allow us to later define\nthe e->size_ field to different sizes without affecting other stuff.\n\n> +\tif (!e->size_valid) {\n> +\t\tunsigned long real_size;\n> +\n> +\t\tif (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n> +\t\t size != real_size)\n> +\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n> +\t}\n\nIf an object that is smaller than 4GB is fed to this function with\nan incorrect size, we happily record it in e->size_ and declare it\nis valid. Wouldn't that be equally grave error as we are catching\nin this block?\n\n> +}\n> +\n> #endif\n"},{"id":"342202","messageId":"CACsJy8AkJJQ4XNszxBsESN_WGOSZ+ExWdcCtn6NA+gW9+-mAqQ@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqsh8wvwwn.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-19T16:54:48Z","receivedAt":"2018-03-19T16:55:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 19, 2018 at 5:43 PM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> +static inline void oe_set_size(struct object_entry *e,\n>> + unsigned long size)\n>> +{\n>> + e->size_ = size;\n>> + e->size_valid = e->size_ == size;\n>\n> A quite similar comment as my earlier one applies here. I wonder if\n> this is easier to read?\n>\n> e->size_valid = fits_in_32bits(size);\n> if (e->size_valid)\n> e->size_ = size;\n>\n> Stepping back a bit in a different tangent,\n>\n> - fits_in_32bits() is a good public name if the helper is about\n> seeing if the given quantity fits in 32bit uint,\n>\n> - but that carves it in stone that our e->size_ *will* be 32bit\n> forever, which is not good.\n>\n> So, it may be a good idea to call it size_cacheable_in_oe(size) or\n> something to ask \"I have this 'size'; is it small enough to fit in\n> the field in the oe, i.e. allow us to cache it, as opposed to having\n> to go back to the object every time?\" Of course, this would declare\n> that the helper can only be used for that particular field, but that\n> is sort of the point of such a change, to allow us to later define\n> the e->size_ field to different sizes without affecting other stuff.\n\nThis is why I do \"size_valid = size_ == size\". In my private build, I\nreduced size_ to less than 32 bits and change the \"fits_in_32bits\"\nfunction to do something like\n\nint fits_in_32bits(unsigned long size)\n{\nstruct object_entry e;\ne.size_ = size;\nreturn e.size_ == size.\n}\n\nwhich makes sure it always works. This spreads the use of \"valid = xx\n== yy\" in more places though. I think if we just limit the use of\nthis expression in a couple access wrappers than it's not so bad.\n\n>> + if (!e->size_valid) {\n>> + unsigned long real_size;\n>> +\n>> + if (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n>> + size != real_size)\n>> + die(\"BUG: 'size' is supposed to be the object size!\");\n>> + }\n>\n> If an object that is smaller than 4GB is fed to this function with\n> an incorrect size, we happily record it in e->size_ and declare it\n> is valid. Wouldn't that be equally grave error as we are catching\n> in this block?\n\nThat adds an extra sha1_object_info() to all objects and it's\nexpensive (I think it's one of the reasons we cache values in\nobject_entry in the first place). I think there are also a few\noccasions we reuse even bad in-pack objects (there are even tests for\nthat) so it's not always safe to die() here.\n-- \nDuy\n"},{"id":"342210","messageId":"CACsJy8BoA7H+5nYigLRVTVAU70=WRiwQV2GkM10xhJWA2onvyA@mail.gmail.com","threadId":"47943","inReplyTo":"87zi37d93w.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 2/7] gc: add --keep-base-pack","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-19T17:26:38Z","receivedAt":"2018-03-19T17:27:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 16, 2018 at 10:05 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n>\n>> +--keep-base-pack::\n>> + All packs except the base pack and those marked with a `.keep`\n>> + files are consolidated into a single pack. The largest pack is\n>> + considered the base pack.\n>> +\n>\n> I wonder if all of this would be less confusing as:\n>\n>> +--keep-biggest-pack::\n>> + All packs except the largest pack and those marked with a `.keep`\n>> + files are consolidated into a single pack.\n>\n> I.e. just skimming these docs I'd expect \"base\" to somehow be the thing\n> that we initially cloned, of course in almost all cases that *is* the\n> largest pack, but not necessarily. So rather than communicate that\n> expectation let's just say largest/biggest?\n\nKeeping the term base pack allows us to change its definition later\n(something else other than \"largest\"). But to be honest I can't see\nwhat else can a base pack(s) be. So unless people object I'm changing\nthis to --keep-biggest-pack (which could take a value to keep \nlargest packs, but I will refrain from doing things we don't need\nright now).\n\n>\n> Maybe I'm the only one who finds this confusing...\n-- \nDuy\n"},{"id":"342222","messageId":"CACsJy8AbkaSLBYqtFiCwFoqRXguy1hTs=XriefZ3WQJjo_sy8Q@mail.gmail.com","threadId":"47943","inReplyTo":"xmqqlgenvs07.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-19T18:45:04Z","receivedAt":"2018-03-19T18:45:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 19, 2018 at 7:29 PM, Junio C Hamano wrote:\n>>>> + if (!e->size_valid) {\n>>>> + unsigned long real_size;\n>>>> +\n>>>> + if (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n>>>> + size != real_size)\n>>>> + die(\"BUG: 'size' is supposed to be the object size!\");\n>>>> + }\n>>>\n>>> If an object that is smaller than 4GB is fed to this function with\n>>> an incorrect size, we happily record it in e->size_ and declare it\n>>> is valid. Wouldn't that be equally grave error as we are catching\n>>> in this block?\n>>\n>> That adds an extra sha1_object_info() to all objects and it's\n>> expensive (I think it's one of the reasons we cache values in\n>> object_entry in the first place). I think there are also a few\n>> occasions we reuse even bad in-pack objects (there are even tests for\n>> that) so it's not always safe to die() here.\n>\n> So what? My point is that I do not see the point in checking if the\n> size is correct on only one side (i.e. size is too big to fit in\n> e->size_) and not the other. If it is worth checking (perhaps under\n> \"#ifndef NDEBUG\" or some other debug option?) then I'd think we\n> should spend cycles for all objects and check.\n\nThere is a difference. For sizes smaller than 2^32, whatever you pass\nto oe_set_size() will be returned by oe_size(), consistently. It does\nnot matter if this size is \"good\" or not. With sizes > 2^32, we make\nthe assumption that this size must be the same as one found in the\nobject database. If it's different, oe_size() will return something\nelse other than oe_set_size() is given. This check here is to make\nsure we do not accidentally let the caller fall into this trap.\n\nYes, it may be a good thing to check anyway even for sizes < 2^32. I'm\na bit uncomfortable doing that though. I was trying to exercise this\ncode the other day by reducing size_ field down to 4 bits, and a\ncouple tests broke but I still don't understand how. It's probably\njust me pushing the limits too hard, not a bug in these changes. But\nit does tell me that I don't understand pack-objects enough to assert\nthat \"all calls to oe_set_size() give good size\".\n-- \nDuy\n"},{"id":"342227","messageId":"xmqqlgenvs07.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"CACsJy8AkJJQ4XNszxBsESN_WGOSZ+ExWdcCtn6NA+gW9+-mAqQ@mail.gmail.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-19T18:29:12Z","receivedAt":"2018-03-19T18:59:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> This is why I do \"size_valid = size_ == size\". In my private build, I\n> reduced size_ to less than 32 bits and change the \"fits_in_32bits\"\n> function to do something like\n>\n> int fits_in_32bits(unsigned long size)\n> {\n> struct object_entry e;\n> e.size_ = size;\n> return e.size_ == size.\n> }\n>\n> which makes sure it always works. This spreads the use of \"valid = xx\n> == yy\" in more places though. I think if we just limit the use of\n> this expression in a couple access wrappers than it's not so bad.\n\nYes, but then we should name the helper so that it is clear that it\nis not about 32-bit but is about the width of e.size_ field.\n>\n>>> + if (!e->size_valid) {\n>>> + unsigned long real_size;\n>>> +\n>>> + if (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n>>> + size != real_size)\n>>> + die(\"BUG: 'size' is supposed to be the object size!\");\n>>> + }\n>>\n>> If an object that is smaller than 4GB is fed to this function with\n>> an incorrect size, we happily record it in e->size_ and declare it\n>> is valid. Wouldn't that be equally grave error as we are catching\n>> in this block?\n>\n> That adds an extra sha1_object_info() to all objects and it's\n> expensive (I think it's one of the reasons we cache values in\n> object_entry in the first place). I think there are also a few\n> occasions we reuse even bad in-pack objects (there are even tests for\n> that) so it's not always safe to die() here.\n\nSo what? My point is that I do not see the point in checking if the\nsize is correct on only one side (i.e. size is too big to fit in\ne->size_) and not the other. If it is worth checking (perhaps under\n\"#ifndef NDEBUG\" or some other debug option?) then I'd think we\nshould spend cycles for all objects and check.\n\n"},{"id":"342228","messageId":"871sgfdgzk.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"CACsJy8BoA7H+5nYigLRVTVAU70=WRiwQV2GkM10xhJWA2onvyA@mail.gmail.com","subject":"Re: [PATCH v3 2/7] gc: add --keep-base-pack","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-19T19:04:31Z","receivedAt":"2018-03-19T19:04:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 19 2018, Duy Nguyen jotted:\n\n> On Fri, Mar 16, 2018 at 10:05 PM, Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> On Fri, Mar 16 2018, Nguyễn Thái Ngọc Duy jotted:\n>>\n>>> +--keep-base-pack::\n>>> + All packs except the base pack and those marked with a `.keep`\n>>> + files are consolidated into a single pack. The largest pack is\n>>> + considered the base pack.\n>>> +\n>>\n>> I wonder if all of this would be less confusing as:\n>>\n>>> +--keep-biggest-pack::\n>>> + All packs except the largest pack and those marked with a `.keep`\n>>> + files are consolidated into a single pack.\n>>\n>> I.e. just skimming these docs I'd expect \"base\" to somehow be the thing\n>> that we initially cloned, of course in almost all cases that *is* the\n>> largest pack, but not necessarily. So rather than communicate that\n>> expectation let's just say largest/biggest?\n>\n> Keeping the term base pack allows us to change its definition later\n> (something else other than \"largest\"). But to be honest I can't see\n> what else can a base pack(s) be. So unless people object I'm changing\n> this to --keep-biggest-pack (which could take a value to keep \n> largest packs, but I will refrain from doing things we don't need\n> right now).\n\nMaybe I've just been reading this differently, but to me the \"base\" pack\nmeans the pack that holds the basis of our history, i.e. the thing we\nfirst cloned. As in the base of the history.\n\nLet's say we have a 100MB pack that we cloned, and someone adds a 200MB\n(uncompressible) binary file to the repo, then we'll have a \"base\" pack\nthat's smaller than the \"largest\" pack.\n\nSo when I was initially reading this series I kept looking for some\ndiscovery of *that* pack, but of course it turned out that it's just\nlooking for the largest pack.\n\nI just think it's best to avoid that confusion since we really mean\nlargest, and maybe in the future it would be legitimate to treat the\n\"base\" pack differently, i.e. as you pull down more updates you're\nlikely to need to be consulting it less and less as time goes on, so\nmaybe we should have some mode to explicitly exclude just *that* pack\neventually. I.e. as an optimization to keep the more relevant stuff in\ncache.\n"},{"id":"342245","messageId":"xmqqy3inu8qe.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"CACsJy8AbkaSLBYqtFiCwFoqRXguy1hTs=XriefZ3WQJjo_sy8Q@mail.gmail.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-19T20:10:49Z","receivedAt":"2018-03-19T20:11:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> There is a difference. For sizes smaller than 2^32, whatever you\n> pass to oe_set_size() will be returned by oe_size(),\n> consistently. It does not matter if this size is \"good\" .... If\n> it's different, oe_size() will return something else other than\n> oe_set_size() is given.\n\nOK, fair enough.\n\n> ... I was trying to exercise this\n> code the other day by reducing size_ field down to 4 bits, and a\n> couple tests broke but I still don't understand how.\n\nOff by one? Two or more copies of the same objects available whose\noe_size() are different?\n\n"},{"id":"342289","messageId":"20180320052827.GA15813@sigill.intra.peff.net","threadId":"47943","inReplyTo":"87fu4ycq7s.fsf@evledraar.gmail.com","subject":"Re: Why does pack-objects use so much memory on incremental packing?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-20T05:28:27Z","receivedAt":"2018-03-20T05:28:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 17, 2018 at 11:05:59PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> Splitting this off into its own thread. Aside from the improvements in\n> your repack memory reduction (20180317141033.21545-1-pclouds@gmail.com)\n> and gc config (20180316192745.19557-1-pclouds@gmail.com) series's I'm\n> wondering why repack takes so much memory to incrementally repack new\n> stuff when you leave out the base pack.\n\nI think it's a combination of a few issues:\n\n 1. We do a complete history traversal, and then cull out objects which\n our filters reject (e.g., things in a .keep pack). So you pay for\n all of the \"struct object\", along with the obj_hash table to look\n them up.\n\n In my measurements of just \"git rev-list --objects --all\", that's\n about 25MB for git.git. Plus a few misc things (pending object\n structs for the traversal, etc).\n\n 2. The delta-base cache used for the traversal is a fixed size. So\n that's going to be 96MB regardless of your repo size.\n\nI measured a total heap usage of 130MB for \"rev-list --objects --all\".\nThat's not 230, but I'm not sure what you're measuring. If it's RSS,\nkeep in mind that includes the mmap'd packfiles, too.\n\nDoing a separate \"rev-list | pack-objects\" should be minorly cheaper\n(although it will still have a similar peak cost, since that memory will\njust be moved to the rev-list process).\n\nIf you _just_ want to pack the loose objects, you could probably do\nsomething like:\n\n find .git/objects/?? -type f |\n tr -d / |\n git pack-objects .git/objects/pack/pack\n git prune-packed\n\nBut you'd get pretty crappy deltas out of that, since the heuristics\nrely on knowing the filenames of trees and blobs (which you can only get\nby walking the graph).\n\nSo you'd do better with something like:\n\n git rev-list --objects $new_tips --not $old_tips |\n git pack-objects .git/objects/pack/pack\n\nbut it's hard to know what \"$old_tips\" should be, unless you recorded it\nlast time you did a full repack.\n\n> But no, it takes around 230MB. But thinking about it a bit further:\n> \n> * This builds on top of existing history, so that needs to be\n> read/consulted\n\nRight, I think this is the main thing.\n\n> * We might be reusing (if not directly, skipping re-comuting) deltas\n> from the existing pack.\n\nI don't think that should matter. We'll reuse deltas if the base is\ngoing into our pack, but otherwise recompute. The delta computation\nitself takes some memory, but it should be fairly constant even for a\nlarge repo (it's really average_blob_size * window_size).\n\nSo I think most of your memory is just going to the traversal stuff.\nRunning:\n\n valgrind --tool=massif git pack-objects --all foo But I get the same result if after cloning I make an orphan branch, and\n> pass all the \"do this as cheaply as possible\" branches I can find down\n> to git-repack:\n> \n> (\n> rm -rf /tmp/git &&\n> git clone git@github.com:git/git.git /tmp/git &&\n> cd /tmp/git &&\n> touch $(ls .git/objects/pack/*pack | sed 's/\\.pack$/.keep/') &&\n> git checkout --orphan new &&\n> git reset --hard &&\n> for i in {1..10}\n> do\n> touch $i &&\n> git add $i &&\n> git commit -m$i\n> done &&\n> git tag -d $(git tag -l) &&\n> /usr/bin/time -f %M git repack -A -d -f -F --window=1 --depth=1\n> )\n> \n> But the memory use barely changes, my first example used 227924 kb, but\n> this one uses 226788.\n\nI think you still had to do the whole history traversal there, because\nyou have existing refs (the \"master\" branch, along with refs/remotes) as\nwell as reflogs.\n\nTry:\n\n git branch -d master\n git remote rm origin\n rm -rf .git/logs\n\nAfter that, the repack uses about 5MB.\n\n> Jeff: Is this something ref islands[1] could be (ab)used to do, or have\n> I misunderstood that concept?\n> \n> 1. https://public-inbox.org/git/20130626051117.GB26755@sigill.intra.peff.net/\n> https://public-inbox.org/git/20160304153359.GA16300@sigill.intra.peff.net/\n> https://public-inbox.org/git/20160809174528.2ydgkhd7ayclat3t@sigill.intra.peff.net/\n\nI think you misunderstood the concept. :)\n\nThey are about disallowing deltas between unrelated islands. They\nactually require _more_ memory, because you have to storage an island\nbitmap for each object (though with some copy-on-write magic, it's not\ntoo bad). But they can never save you memory, since reused deltas are\nalways cheaper than re-finding new ones.\n\n-Peff\n"},{"id":"342345","messageId":"20180320180806.GA16521@duynguyen.home","threadId":"47943","inReplyTo":"xmqqy3inu8qe.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-20T18:08:07Z","receivedAt":"2018-03-20T18:08:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 19, 2018 at 01:10:49PM -0700, Junio C Hamano wrote:\n> > ... I was trying to exercise this\n> > code the other day by reducing size_ field down to 4 bits, and a\n> > couple tests broke but I still don't understand how.\n> \n> Off by one? Two or more copies of the same objects available whose\n> oe_size() are different?\n> \n\nNo. I did indeed not understand pack-objects enough :)\n\nThis \"size\" field contains the delta size if the in-pack object is a\ndelta. So blindly falling back to object_sha1_info() which returns the\ncanonical object size is definitely wrong. Please eject the series\nfrom 'pu' until I fix this. The bug won't likely affect anyone (since\nthey must have 4GB+ objects to trigger it) but better safe than sorry.\n\nBTW can you apply this patch? This broken && chain made me think the\nproblem was in the next test. It would have saved me lots of time if I\nsaw this \"BUG\" line coming from the previous test.\n\n-- 8< --\nSubject: [PATCH] t9300: fix broken && chain\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t9300-fast-import.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex e4d06accc4..e2a0ae4075 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -348,7 +348,7 @@ test_expect_success 'B: accept branch name \"TEMP_TAG\"' '\n \tINPUT_END\n \n \ttest_when_finished \"rm -f .git/TEMP_TAG\n-\t\tgit gc\n+\t\tgit gc &&\n \t\tgit prune\" &&\n \tgit fast-import wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> +static inline void oe_set_size(struct object_entry *e,\n>> + unsigned long size)\n>> +{\n>> + e->size_ = size;\n>> + e->size_valid = e->size_ == size;\n>\n> A quite similar comment as my earlier one applies here. I wonder if\n> this is easier to read?\n>\n> e->size_valid = fits_in_32bits(size);\n> if (e->size_valid)\n> e->size_ = size;\n\nI wonder if wrapping this \"==\" with something like this would help readability?\n\n#define truncated(a,b) (a) != (b)\n\nThen we could write\n\ne->size_valid = !truncated(e->size_, size);\n-- \nDuy\n"},{"id":"342349","messageId":"xmqq370usj3q.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180320180806.GA16521@duynguyen.home","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-20T18:22:01Z","receivedAt":"2018-03-20T18:22:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> This \"size\" field contains the delta size if the in-pack object is a\n> delta. So blindly falling back to object_sha1_info() which returns the\n> canonical object size is definitely wrong.\n\nYup. Also we need to be careful when going back to the packfile to\nread the size in question. A different packfile that has the same\nobject may have delta that was constructed differently and of wrong\nsize.\n\n> Please eject the series\n> from 'pu' until I fix this. The bug won't likely affect anyone (since\n> they must have 4GB+ objects to trigger it) but better safe than sorry.\n\n> BTW can you apply this patch? This broken && chain made me think the\n> problem was in the next test. It would have saved me lots of time if I\n> saw this \"BUG\" line coming from the previous test.\n\nThanks, will do.\n\n>\n> -- 8< --\n> Subject: [PATCH] t9300: fix broken && chain\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> t/t9300-fast-import.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index e4d06accc4..e2a0ae4075 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -348,7 +348,7 @@ test_expect_success 'B: accept branch name \"TEMP_TAG\"' '\n> \tINPUT_END\n> \n> \ttest_when_finished \"rm -f .git/TEMP_TAG\n> -\t\tgit gc\n> +\t\tgit gc &&\n> \t\tgit prune\" &&\n> \tgit fast-import \ttest -f .git/TEMP_TAG &&\n> @@ -365,7 +365,7 @@ test_expect_success 'B: accept empty committer' '\n> \tINPUT_END\n> \n> \ttest_when_finished \"git update-ref -d refs/heads/empty-committer-1\n> -\t\tgit gc\n> +\t\tgit gc &&\n> \t\tgit prune\" &&\n> \tgit fast-import \tout=$(git fsck) &&\n"},{"id":"342392","messageId":"20180321080343.GA25537@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180320180806.GA16521@duynguyen.home","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-21T08:03:44Z","receivedAt":"2018-03-21T08:03:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 20, 2018 at 07:08:07PM +0100, Duy Nguyen wrote:\n\n> BTW can you apply this patch? This broken && chain made me think the\n> problem was in the next test. It would have saved me lots of time if I\n> saw this \"BUG\" line coming from the previous test.\n> \n> -- 8< --\n> Subject: [PATCH] t9300: fix broken && chain\n> \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> t/t9300-fast-import.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index e4d06accc4..e2a0ae4075 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -348,7 +348,7 @@ test_expect_success 'B: accept branch name \"TEMP_TAG\"' '\n> \tINPUT_END\n> \n> \ttest_when_finished \"rm -f .git/TEMP_TAG\n> -\t\tgit gc\n> +\t\tgit gc &&\n> \t\tgit prune\" &&\n\nThe &&-chain is broken from the first command, too. It's \"rm -f\", which\nis not that big a deal, but...\n\n> @@ -365,7 +365,7 @@ test_expect_success 'B: accept empty committer' '\n> \tINPUT_END\n> \n> \ttest_when_finished \"git update-ref -d refs/heads/empty-committer-1\n> -\t\tgit gc\n> +\t\tgit gc &&\n> \t\tgit prune\" &&\n\nSame here, but we probably care more about noticing update-ref failure.\n\n-Peff\n"},{"id":"342393","messageId":"20180321082441.GB25537@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-21T08:24:41Z","receivedAt":"2018-03-21T08:24:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> v6 fixes the one optimization that I just couldn't get right, fixes\n> two off-by-one error messages and a couple commit message update\n> (biggest change is in 11/11 to record some numbers from AEvar)\n\nI was traveling during some of the earlier rounds, so I finally got a\nchance to take a look at this.\n\nI hate to be a wet blanket, but am I the only one who is wondering\nwhether the tradeoffs is worth it? 8% memory reduction doesn't seem\nmind-bogglingly good, and I'm concerned about two things:\n\n 1. The resulting code is harder to read and reason about (things like\n the DELTA() macros), and seems a lot more brittle (things like the\n new size_valid checks).\n\n 2. There are lots of new limits. Some of these are probably fine\n (e.g., the cacheable delta size), but things like the\n number-of-packs limit don't have very good user-facing behavior.\n Yes, having that many packs is insane, but that's going to be small\n consolation to somebody whose automated maintenance program now\n craps out at 16k packs, when it previously would have just worked\n to fix the situation.\n\nSaving 8% is nice, but the number of objects in linux.git grew over 12%\nin the last year. So you've bought yourself 8 months before the problem\nis back. Is it worth making these changes that we'll have to deal with\nfor many years to buy 8 months of memory savings?\n\nI think ultimately to work on low-memory machines we'll need a\nfundamentally different approach that scales with the objects since the\nlast pack, and not with the complete history.\n\n-Peff\n"},{"id":"342411","messageId":"CACsJy8DkF3TpTGKp5MdS1ApC8Yj6FOL5uAg5U__PjBAQY1vdNw@mail.gmail.com","threadId":"47943","inReplyTo":"20180321082441.GB25537@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-21T15:59:19Z","receivedAt":"2018-03-21T15:59:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote:\n> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> v6 fixes the one optimization that I just couldn't get right, fixes\n>> two off-by-one error messages and a couple commit message update\n>> (biggest change is in 11/11 to record some numbers from AEvar)\n>\n> I was traveling during some of the earlier rounds, so I finally got a\n> chance to take a look at this.\n>\n> I hate to be a wet blanket, but am I the only one who is wondering\n> whether the tradeoffs is worth it? 8% memory reduction doesn't seem\n> mind-bogglingly good,\n\nAEvar measured RSS. If we count objects[] array alone, the saving is\n40% (136 bytes per entry down to 80). Some is probably eaten up by\nmmap in rss.\n\n> and I'm concerned about two things:\n>\n> 1. The resulting code is harder to read and reason about (things like\n> the DELTA() macros), and seems a lot more brittle (things like the\n> new size_valid checks).\n>\n> 2. There are lots of new limits. Some of these are probably fine\n> (e.g., the cacheable delta size), but things like the\n> number-of-packs limit don't have very good user-facing behavior.\n> Yes, having that many packs is insane, but that's going to be small\n> consolation to somebody whose automated maintenance program now\n> craps out at 16k packs, when it previously would have just worked\n> to fix the situation.\n>\n> Saving 8% is nice, but the number of objects in linux.git grew over 12%\n> in the last year. So you've bought yourself 8 months before the problem\n> is back. Is it worth making these changes that we'll have to deal with\n> for many years to buy 8 months of memory savings?\n\nWell, with 40% it buys us a couple more months. The object growth\naffects rev-list --all too so the actual \"good months\" is probably not\nsuper far from 8 months.\n\nIs it worth saving? I don't know. I raised the readability point from\nthe very first patch and if people believe it makes it much harder to\nread, then no it's not worth it.\n\nWhile pack-objects is simple from the functionality point of view, it\nhas received lots of optimizations and to me is quite fragile.\nReadability does count in this code. Fortunately it still looks quite\nok to me with this series applied (but then it's subjective)\n\nAbout the 16k limit (and some other limits as well), I'm making these\npatches with the assumption that large scale deployment probably will\ngo with custom builds anyway. Adjusting the limits back should be\nquite easy while we can still provide reasonable defaults for most\npeople.\n\n> I think ultimately to work on low-memory machines we'll need a\n> fundamentally different approach that scales with the objects since the\n> last pack, and not with the complete history.\n\nAbsolutely. Which is covered in a separate \"gc --auto\" series. Some\nmemory reduction here may be still nice to have though. Even on beefy\nmachine, memory can still be reused somewhere other than wasted in\nunused bits.\n-- \nDuy\n"},{"id":"342414","messageId":"CACsJy8BErdP31AGxSDBMQ-a60WVMSVsUjaJj5xscFGUbgyCWwA@mail.gmail.com","threadId":"47943","inReplyTo":"20180321080343.GA25537@sigill.intra.peff.net","subject":"Re: [PATCH v6 09/11] pack-objects: shrink size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-21T16:12:09Z","receivedAt":"2018-03-21T16:12:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 21, 2018 at 9:03 AM, Jeff King wrote:\n> On Tue, Mar 20, 2018 at 07:08:07PM +0100, Duy Nguyen wrote:\n>\n>> BTW can you apply this patch? This broken && chain made me think the\n>> problem was in the next test. It would have saved me lots of time if I\n>> saw this \"BUG\" line coming from the previous test.\n>>\n>> -- 8< --\n>> Subject: [PATCH] t9300: fix broken && chain\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>> t/t9300-fast-import.sh | 4 ++--\n>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n>> index e4d06accc4..e2a0ae4075 100755\n>> --- a/t/t9300-fast-import.sh\n>> +++ b/t/t9300-fast-import.sh\n>> @@ -348,7 +348,7 @@ test_expect_success 'B: accept branch name \"TEMP_TAG\"' '\n>> INPUT_END\n>>\n>> test_when_finished \"rm -f .git/TEMP_TAG\n>> - git gc\n>> + git gc &&\n>> git prune\" &&\n>\n> The &&-chain is broken from the first command, too. It's \"rm -f\", which\n> is not that big a deal, but...\n>\n>> @@ -365,7 +365,7 @@ test_expect_success 'B: accept empty committer' '\n>> INPUT_END\n>>\n>> test_when_finished \"git update-ref -d refs/heads/empty-committer-1\n>> - git gc\n>> + git gc &&\n>> git prune\" &&\n>\n> Same here, but we probably care more about noticing update-ref failure.\n\nYes. I wasn't sure if that update-ref could fail but did not check\nsince this was a side issue for me.\n-- \nDuy\n"},{"id":"342416","messageId":"87vadpxv27.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"CACsJy8DkF3TpTGKp5MdS1ApC8Yj6FOL5uAg5U__PjBAQY1vdNw@mail.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-21T16:17:04Z","receivedAt":"2018-03-21T16:17:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 21 2018, Duy Nguyen wrote:\n\n> On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote:\n>> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:\n>>\n>>> v6 fixes the one optimization that I just couldn't get right, fixes\n>>> two off-by-one error messages and a couple commit message update\n>>> (biggest change is in 11/11 to record some numbers from AEvar)\n>>\n>> I was traveling during some of the earlier rounds, so I finally got a\n>> chance to take a look at this.\n>>\n>> I hate to be a wet blanket, but am I the only one who is wondering\n>> whether the tradeoffs is worth it? 8% memory reduction doesn't seem\n>> mind-bogglingly good,\n>\n> AEvar measured RSS. If we count objects[] array alone, the saving is\n> 40% (136 bytes per entry down to 80). Some is probably eaten up by\n> mmap in rss.\n\nYeah, sorry about spreading that confusion.\n\n>> and I'm concerned about two things:\n>>\n>> 1. The resulting code is harder to read and reason about (things like\n>> the DELTA() macros), and seems a lot more brittle (things like the\n>> new size_valid checks).\n>>\n>> 2. There are lots of new limits. Some of these are probably fine\n>> (e.g., the cacheable delta size), but things like the\n>> number-of-packs limit don't have very good user-facing behavior.\n>> Yes, having that many packs is insane, but that's going to be small\n>> consolation to somebody whose automated maintenance program now\n>> craps out at 16k packs, when it previously would have just worked\n>> to fix the situation.\n>>\n>> Saving 8% is nice, but the number of objects in linux.git grew over 12%\n>> in the last year. So you've bought yourself 8 months before the problem\n>> is back. Is it worth making these changes that we'll have to deal with\n>> for many years to buy 8 months of memory savings?\n>\n> Well, with 40% it buys us a couple more months. The object growth\n> affects rev-list --all too so the actual \"good months\" is probably not\n> super far from 8 months.\n>\n> Is it worth saving? I don't know. I raised the readability point from\n> the very first patch and if people believe it makes it much harder to\n> read, then no it's not worth it.\n>\n> While pack-objects is simple from the functionality point of view, it\n> has received lots of optimizations and to me is quite fragile.\n> Readability does count in this code. Fortunately it still looks quite\n> ok to me with this series applied (but then it's subjective)\n>\n> About the 16k limit (and some other limits as well), I'm making these\n> patches with the assumption that large scale deployment probably will\n> go with custom builds anyway. Adjusting the limits back should be\n> quite easy while we can still provide reasonable defaults for most\n> people.\n>\n>> I think ultimately to work on low-memory machines we'll need a\n>> fundamentally different approach that scales with the objects since the\n>> last pack, and not with the complete history.\n>\n> Absolutely. Which is covered in a separate \"gc --auto\" series. Some\n> memory reduction here may be still nice to have though. Even on beefy\n> machine, memory can still be reused somewhere other than wasted in\n> unused bits.\n\nFWIW I've been running a combination of these two at work (also keeping\nthe big pack), and they've had a sizable impact on packing our monorepo,\non one of our dev boxes on a real-world checkout with a combo of the\n\"base\" pack and other packs + loose objects, as measured by\n/usr/bin/time\n\n * Reduction in user time by 37%\n * Reduction in system time by 84%\n * Reduction in RSS by 61%\n * Reduction in page faults by 58% & 94% (time(1) reports two different numbers)\n * Reduction in the I of I/O by 58%\n * Reduction in the O of I/O by 94%\n"},{"id":"342418","messageId":"CACsJy8BmCHnnTQgdQjvv2msqg3XL12oeLxy3RPTTdGJrUSarjg@mail.gmail.com","threadId":"47943","inReplyTo":"87vadpxv27.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-21T16:22:42Z","receivedAt":"2018-03-21T16:24:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 21, 2018 at 5:17 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>>> I think ultimately to work on low-memory machines we'll need a\n>>> fundamentally different approach that scales with the objects since the\n>>> last pack, and not with the complete history.\n>>\n>> Absolutely. Which is covered in a separate \"gc --auto\" series. Some\n>> memory reduction here may be still nice to have though. Even on beefy\n>> machine, memory can still be reused somewhere other than wasted in\n>> unused bits.\n>\n> FWIW I've been running a combination of these two at work (also keeping\n> the big pack), and they've had a sizable impact on packing our monorepo,\n> on one of our dev boxes on a real-world checkout with a combo of the\n> \"base\" pack and other packs + loose objects, as measured by\n> /usr/bin/time\n>\n> * Reduction in user time by 37%\n> * Reduction in system time by 84%\n> * Reduction in RSS by 61%\n> * Reduction in page faults by 58% & 94% (time(1) reports two different numbers)\n> * Reduction in the I of I/O by 58%\n> * Reduction in the O of I/O by 94%\n\nThe keeping big pack changes very likely contributes to most of this\nreduction, so just to be clear these numbers can't be be used as an\nargument in favor of this pack-objects series (but otherwise, wow! I\nguess I need to finish up the gc series soon, then start the external\nrev-list work to reduce even more ;-)\n-- \nDuy\n"},{"id":"342419","messageId":"87tvt9xuel.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180321082441.GB25537@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-21T16:31:14Z","receivedAt":"2018-03-21T16:31:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 21 2018, Jeff King wrote:\n\n> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> v6 fixes the one optimization that I just couldn't get right, fixes\n>> two off-by-one error messages and a couple commit message update\n>> (biggest change is in 11/11 to record some numbers from AEvar)\n>\n> [...]Yes, having that many packs is insane, but that's going to be\n> small consolation to somebody whose automated maintenance program now\n> craps out at 16k packs, when it previously would have just worked to\n> fix the situation[...]\n\nThat's going to be super rare (and probably nonexisting) edge case, but\n(untested) I wonder if something like this on top would alleviate your\nconcerns, i.e. instead of dying we just take the first N packs up to our\nlimit:\n\n diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n index 4406af640f..49d467ab2a 100644\n --- a/builtin/pack-objects.c\n +++ b/builtin/pack-objects.c\n @@ -1065,8 +1065,9 @@ static int want_object_in_pack(const struct object_id *oid,\n\n want = 1;\n done:\n - if (want && *found_pack && !(*found_pack)->index)\n - oe_add_pack(&to_pack, *found_pack);\n + if (want && *found_pack && !(*found_pack)->index) {\n + if (oe_add_pack(&to_pack, *found_pack) == -1)\n + return 0;\n\n return want;\n }\n diff --git a/pack-objects.h b/pack-objects.h\n index 9f8e450e19..50ed2028fb 100644\n --- a/pack-objects.h\n +++ b/pack-objects.h\n @@ -171,15 +171,17 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n pack->in_pack_pos[e - pack->objects] = pos;\n }\n\n -static inline unsigned int oe_add_pack(struct packing_data *pack,\n +static inline int oe_add_pack(struct packing_data *pack,\n struct packed_git *p)\n {\n - if (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n - die(_(\"too many packs to handle in one go. \"\n - \"Please add .keep files to exclude\\n\"\n - \"some pack files and keep the number \"\n - \"of non-kept files below %d.\"),\n + if (pack->in_pack_count >= (1 << OE_IN_PACK_BITS)) {\n + warning(_(\"Too many packs to handle in one go. \"\n + \"Ran into the limit of %d.\\n\"\n + \"Limping along by pretending packs beyond that\"\n + \"number have *.keep!\"),\n 1 << OE_IN_PACK_BITS);\n + return -1;\n + }\n if (p) {\n if (p->index > 0)\n die(\"BUG: this packed is already indexed\");\n"},{"id":"342424","messageId":"20180321164629.GA27584@duynguyen","threadId":"47943","inReplyTo":"CACsJy8DkF3TpTGKp5MdS1ApC8Yj6FOL5uAg5U__PjBAQY1vdNw@mail.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-21T16:46:29Z","receivedAt":"2018-03-21T16:46:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:\n> About the 16k limit (and some other limits as well), I'm making these\n> patches with the assumption that large scale deployment probably will\n> go with custom builds anyway. Adjusting the limits back should be\n> quite easy while we can still provide reasonable defaults for most\n> people.\n\nAnd we could even do something like this to make custom builds\neasier. Some more gluing is needed so you can set this from config.mak\nbut you get the idea. This removes all limits set by this\nseries. Readability in pack-objects.c and object_entry struct\ndeclaration is still a concern though.\n\n-- 8< --\ndiff --git a/pack-objects.h b/pack-objects.h\nindex af40211105..b6e84c9b48 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,10 +2,17 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#ifdef PACK_OBJECTS_BIG_MEMORY\n+#define OE_DEPTH_BITS\t\t31\n+/* OE_IN_PACK_BITS is not defined */\n+#define OE_Z_DELTA_BITS\t\t32\n+#define OE_DELTA_SIZE_BITS\t32\n+#else\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t14\n #define OE_Z_DELTA_BITS\t\t16\n #define OE_DELTA_SIZE_BITS\t31\n+#endif\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -82,7 +89,11 @@ struct object_entry {\n \t\t\t\t */\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n+#ifdef PACK_OBJECTS_BIG_MEMORY\n+\tstruct packed_git *in_pack; /* already in pack */\n+#else\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+#endif\n \tunsigned size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_valid:1;\n@@ -112,7 +123,9 @@ struct packing_data {\n \n \tunsigned int *in_pack_pos;\n \tint in_pack_count;\n+#ifndef PACK_OBJECTS_BIG_MEMORY\n \tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n+#endif\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -174,6 +187,9 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n static inline unsigned int oe_add_pack(struct packing_data *pack,\n \t\t\t\t struct packed_git *p)\n {\n+#ifdef PACK_OBJECTS_BIG_MEMORY\n+\treturn 0;\n+#else\n \tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n \t\tdie(_(\"too many packs to handle in one go. \"\n \t\t \"Please add .keep files to exclude\\n\"\n@@ -187,22 +203,31 @@ static inline unsigned int oe_add_pack(struct packing_data *pack,\n \t}\n \tpack->in_pack[pack->in_pack_count] = p;\n \treturn pack->in_pack_count++;\n+#endif\n }\n \n static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n \t\t\t\t\t const struct object_entry *e)\n {\n+#ifdef PACK_OBJECTS_BIG_MEMORY\n+\treturn e->in_pack;\n+#else\n \treturn pack->in_pack[e->in_pack_idx];\n+#endif\n \n }\n \n static inline void oe_set_in_pack(struct object_entry *e,\n \t\t\t\t struct packed_git *p)\n {\n+#ifdef PACK_OBJECTS_BIG_MEMORY\n+\te->in_pack = p;\n+#else\n \tif (p->index <= 0)\n \t\tdie(\"BUG: found_pack should be NULL \"\n \t\t \"instead of having non-positive index\");\n \te->in_pack_idx = p->index;\n+#endif\n \n }\n \n-- 8< --\n"},{"id":"342425","messageId":"xmqq370tqsjk.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"87tvt9xuel.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-21T16:53:19Z","receivedAt":"2018-03-21T16:53:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> That's going to be super rare (and probably nonexisting) edge case, but\n> (untested) I wonder if something like this on top would alleviate your\n> concerns, i.e. instead of dying we just take the first N packs up to our\n> limit:\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 4406af640f..49d467ab2a 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1065,8 +1065,9 @@ static int want_object_in_pack(const struct object_id *oid,\n>\n> want = 1;\n> done:\n> - if (want && *found_pack && !(*found_pack)->index)\n> - oe_add_pack(&to_pack, *found_pack);\n> + if (want && *found_pack && !(*found_pack)->index) {\n> + if (oe_add_pack(&to_pack, *found_pack) == -1)\n> + return 0;\n>\n> return want;\n> }\n\nIt is probably a small first step in the right direction, but we'd\nneed to communicate which packs we ignored with this logic to the\ncalling program. I offhand do not know how we would handle the \"-d\"\npart of \"repack -a -d\" without it.\n\n"},{"id":"342426","messageId":"CACsJy8CWkm_Xn2c0f46jUdD5dH7NUzTqyu6TrRkGKTS8WRCe3Q@mail.gmail.com","threadId":"47943","inReplyTo":"xmqq370tqsjk.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-21T17:00:46Z","receivedAt":"2018-03-21T17:01:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 21, 2018 at 5:53 PM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> That's going to be super rare (and probably nonexisting) edge case, but\n>> (untested) I wonder if something like this on top would alleviate your\n>> concerns, i.e. instead of dying we just take the first N packs up to our\n>> limit:\n>>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index 4406af640f..49d467ab2a 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -1065,8 +1065,9 @@ static int want_object_in_pack(const struct object_id *oid,\n>>\n>> want = 1;\n>> done:\n>> - if (want && *found_pack && !(*found_pack)->index)\n>> - oe_add_pack(&to_pack, *found_pack);\n>> + if (want && *found_pack && !(*found_pack)->index) {\n>> + if (oe_add_pack(&to_pack, *found_pack) == -1)\n>> + return 0;\n>>\n>> return want;\n>> }\n>\n> It is probably a small first step in the right direction, but we'd\n> need to communicate which packs we ignored with this logic to the\n> calling program. I offhand do not know how we would handle the \"-d\"\n> part of \"repack -a -d\" without it.\n\nrepack will delete all the packs except ones with .keep files and ones\ncreated by pack-objects. So this change alone is not enough. I think I\ndid mention that we could make this work by making repack run\npack-objects multiple times. But I did not do it because I did not\nthink it could really happen.\n-- \nDuy\n"},{"id":"342442","messageId":"xmqqd0zxp7kn.fsf@gitster-ct.c.googlers.com","threadId":"47943","inReplyTo":"20180321164629.GA27584@duynguyen","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-21T19:11:36Z","receivedAt":"2018-03-21T19:11:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> And we could even do something like this to make custom builds\n> easier. Some more gluing is needed so you can set this from config.mak\n> but you get the idea. This removes all limits set by this\n> series.\n\nYes, we _could_, but it would mean we would have many variants of\nthe codepath that is pretty crucial to the integrity of the data we\nkeep in the repository, all of which must pretty much be bug-free.\n\n> Readability in pack-objects.c and object_entry struct declaration\n> is still a concern though.\n\nYup, a change like this does not change the readability; personally,\nI do not think the original is _too_ bad, though.\n\n"},{"id":"342491","messageId":"20180322080740.GA28907@sigill.intra.peff.net","threadId":"47943","inReplyTo":"87tvt9xuel.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-22T08:07:40Z","receivedAt":"2018-03-22T08:07:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > [...]Yes, having that many packs is insane, but that's going to be\n> > small consolation to somebody whose automated maintenance program now\n> > craps out at 16k packs, when it previously would have just worked to\n> > fix the situation[...]\n> \n> That's going to be super rare (and probably nonexisting) edge case, but\n> (untested) I wonder if something like this on top would alleviate your\n> concerns, i.e. instead of dying we just take the first N packs up to our\n> limit:\n\nI wish you were right about the rarity, but it's unfortunately something\nI have seen multiple times in the wild (and why I spent time optimizing\nthe many-packs case for pack-objects). Unfortunately I don't know how\noften it actually comes up, because in theory running \"git repack\"\ncleans it up without further ado. But after these patches, not so much.\n\nI'll admit that my experiences aren't necessarily typical of most git\nusers. But I wouldn't be surprised if other people hosting their own\nrepositories run into this, too (e.g., somebody pushing in a loop,\nauto-gc disabled or clogged by something silly like the \"too many loose\nobjects\" warning).\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 4406af640f..49d467ab2a 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1065,8 +1065,9 @@ static int want_object_in_pack(const struct object_id *oid,\n> \n> want = 1;\n> done:\n> - if (want && *found_pack && !(*found_pack)->index)\n> - oe_add_pack(&to_pack, *found_pack);\n> + if (want && *found_pack && !(*found_pack)->index) {\n> + if (oe_add_pack(&to_pack, *found_pack) == -1)\n> + return 0;\n\nSomething like this does seem like a much better fallback, as we'd make\nforward progress instead of aborting (and exacerbating whatever caused\nthe packs to stack up in the first place).\n\nI think the patch as-is does not work, though. You say \"oops, too many\npacks\" and so the \"yes we want this object\" return becomes \"no, we do\nnot want it\". And it is not included in the resulting packfile.\n\nBut what happens after that? After pack-objects finishes, we return to\n\"git repack\", which assumes that pack-objects packed everything it was\ntold to. And with \"-d\", it then _deletes_ the old packs, knowing that\nanything of value was copied to the new pack. So with this patch, we'd\ncorrupt the repository if this code is ever hit.\n\nYou'd need some way to report back to \"git repack\" that the pack was\nomitted. Or probably more sensibly, you'd need \"git repack\" to count up\nthe packs and make sure that it marks anybody beyond the limit manually\nas .keep (presumably using Duy's new command-line option rather than\nactually writing a file).\n\n-Peff\n"},{"id":"342492","messageId":"CACsJy8AF0FYysw48Y1CKdAHs_zWN8raYaGXrKQ9OmeKS-XujLQ@mail.gmail.com","threadId":"47943","inReplyTo":"20180322080740.GA28907@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-22T08:23:42Z","receivedAt":"2018-03-22T08:24:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 22, 2018 at 9:07 AM, Jeff King wrote:\n> On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > [...]Yes, having that many packs is insane, but that's going to be\n>> > small consolation to somebody whose automated maintenance program now\n>> > craps out at 16k packs, when it previously would have just worked to\n>> > fix the situation[...]\n>>\n>> That's going to be super rare (and probably nonexisting) edge case, but\n>> (untested) I wonder if something like this on top would alleviate your\n>> concerns, i.e. instead of dying we just take the first N packs up to our\n>> limit:\n>\n> I wish you were right about the rarity, but it's unfortunately something\n> I have seen multiple times in the wild (and why I spent time optimizing\n> the many-packs case for pack-objects). Unfortunately I don't know how\n> often it actually comes up, because in theory running \"git repack\"\n> cleans it up without further ado. But after these patches, not so much.\n\nIt's good to know this case is real and I can start to fix it\n(assuming that the other concern about readability will not stop this\nseries).\n\nI think I'll try to fix this without involving repack. pack-objects\ncan produce multiple packs, so if we have more than 16k pack files, we\nproduce one new pack per 16k old ones.\n-- \nDuy\n"},{"id":"342494","messageId":"20180322093212.GB28907@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8DkF3TpTGKp5MdS1ApC8Yj6FOL5uAg5U__PjBAQY1vdNw@mail.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-22T09:32:12Z","receivedAt":"2018-03-22T09:32:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:\n\n> > I hate to be a wet blanket, but am I the only one who is wondering\n> > whether the tradeoffs is worth it? 8% memory reduction doesn't seem\n> > mind-bogglingly good,\n> \n> AEvar measured RSS. If we count objects[] array alone, the saving is\n> 40% (136 bytes per entry down to 80). Some is probably eaten up by\n> mmap in rss.\n\nMeasuring actual heap usage with massif, I get before/after peak heaps\nof 1728 and 1346MB respectively when repacking linux.git. So that's ~22%\nsavings overall.\n\nOf the used heap after your patches:\n\n - ~40% of that is from packlist_alloc()\n - ~17% goes to \"struct object\"\n - ~10% for the object.c hash table to store all the \"struct object\"\n - ~7% goes to the delta cache\n - ~7% goes to the pack revindex (actually, there's a duplicate 7%\n there, too; I think our peak is when we're sorting the revindex\n and have to keep two copies in memory at once)\n - ~5% goes to the packlist_find() hash table\n - ~3.5% for the get_object_details() sorting list (this is only held\n\t for a minute, but again, our peak comes during this sort, which\n\t in turn loads the revindex)\n\nSo 27% of the total heap goes away if you switch to a separate rev-list.\nThough it's mostly just going to a different process, it does help peak\nbecause that process would have exited by the time we get to the\nrevindex bits.\n\nI suspect you could get the same effect by just teaching pack-objects to\nclear obj_hash and all of the allocated object structs. I think that\nshould be safe to do as long as we clear _all_ of the objects, so there\nare no dangling pointers.\n\n> About the 16k limit (and some other limits as well), I'm making these\n> patches with the assumption that large scale deployment probably will\n> go with custom builds anyway. Adjusting the limits back should be\n> quite easy while we can still provide reasonable defaults for most\n> people.\n\nI think this 16k limit is the thing I _most_ dislike about the series.\nIf we could tweak that case such that we always made forward progress, I\nthink I'd be a lot less nervous. I responded elsewhere in the thread\n(before seeing that both Junio and you seemed aware of the issues ;) ),\nbut I think it would be acceptable to have git-repack enforce the limit.\n\nThat would still mean you could get into a broken state for serving\nfetches, but you could at least get out of it by running \"git repack\".\n\n-Peff\n"},{"id":"342495","messageId":"20180322094649.GA32698@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180322093212.GB28907@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-22T09:46:49Z","receivedAt":"2018-03-22T09:46:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 22, 2018 at 05:32:12AM -0400, Jeff King wrote:\n\n> So 27% of the total heap goes away if you switch to a separate rev-list.\n> Though it's mostly just going to a different process, it does help peak\n> because that process would have exited by the time we get to the\n> revindex bits.\n> \n> I suspect you could get the same effect by just teaching pack-objects to\n> clear obj_hash and all of the allocated object structs. I think that\n> should be safe to do as long as we clear _all_ of the objects, so there\n> are no dangling pointers.\n\nThe patch below tries that. It's kind of hacky, but it drops my peak\nheap for packing linux.git from 1336MB to 1129MB.\n\nThat's not quite as exciting as 27%, because it just moves our peak\nearlier, to when we do have all of the object structs in memory (so the\nsavings are really just that we're not holding the revindex, etc at the\nsame time as the object structs).\n\nBut we also hold that peak for a lot shorter period, because we drop the\nmemory before we do any delta compression (which itself can be memory\nhungry[1]), and don't hold onto it during the write phase (which can be\nnetwork-limited when serving a fetch). So during that write phase we're\nholding only ~900MB instead of ~1250MB.\n\n-Peff\n\n[1] All of my timings are on noop repacks of a single pack, so there's\n no actual delta compression. On average, it will use something like\n \"nr_threads * window * avg_blob_size\". For a \"normal\" repo, that's\n only a few megabytes. But the peak will depend on the large blobs,\n so it could have some outsize cases. I don't know if it's worth\n worrying about too much for this analysis.\n\n---\nHere's the patch. It's probably asking for trouble to have this kind of\nclearing interface, as a surprising number of things may hold onto\npointers to objects (see the comment below about the bitmap code).\n\nSo maybe the separate process is less insane.\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacd..50d444a3b0 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -30,15 +30,23 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* book-keeping for clearing */\n+\tvoid *start;\n+\tstruct alloc_state *prev;\n };\n \n-static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n+static inline void *alloc_node(struct alloc_state **sp, size_t node_size)\n {\n+\tstruct alloc_state *s = *sp;\n \tvoid *ret;\n \n-\tif (!s->nr) {\n+\tif (!s || !s->nr) {\n+\t\ts = xmalloc(sizeof(*s));\n \t\ts->nr = BLOCKING;\n-\t\ts->p = xmalloc(BLOCKING * node_size);\n+\t\ts->start = s->p = xmalloc(BLOCKING * node_size);\n+\t\ts->prev = *sp;\n+\t\t*sp = s;\n \t}\n \ts->nr--;\n \ts->count++;\n@@ -48,7 +56,7 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n+static struct alloc_state *blob_state;\n void *alloc_blob_node(void)\n {\n \tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n@@ -56,7 +64,7 @@ void *alloc_blob_node(void)\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n+static struct alloc_state *tree_state;\n void *alloc_tree_node(void)\n {\n \tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n@@ -64,7 +72,7 @@ void *alloc_tree_node(void)\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n+static struct alloc_state *tag_state;\n void *alloc_tag_node(void)\n {\n \tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n@@ -72,7 +80,7 @@ void *alloc_tag_node(void)\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n+static struct alloc_state *object_state;\n void *alloc_object_node(void)\n {\n \tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n@@ -80,7 +88,7 @@ void *alloc_object_node(void)\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n+static struct alloc_state *commit_state;\n \n unsigned int alloc_commit_index(void)\n {\n@@ -103,7 +111,7 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, name##_state->count, name##_state->count * sizeof(type) >> 10)\n \n void alloc_report(void)\n {\n@@ -113,3 +121,22 @@ void alloc_report(void)\n \tREPORT(tag, struct tag);\n \tREPORT(object, union any_object);\n }\n+\n+static void alloc_clear(struct alloc_state **sp)\n+{\n+\twhile (*sp) {\n+\t\tstruct alloc_state *s = *sp;\n+\t\t*sp = s->prev;\n+\t\tfree(s->start);\n+\t\tfree(s);\n+\t}\n+}\n+\n+void alloc_clear_all(void)\n+{\n+\talloc_clear(&blob_state);\n+\talloc_clear(&tree_state);\n+\talloc_clear(&commit_state);\n+\talloc_clear(&tag_state);\n+\talloc_clear(&object_state);\n+}\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4406af640f..7ba8ab07a3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2959,6 +2959,13 @@ static void get_object_list(int ac, const char **av)\n \t\t\t\t record_recent_object, NULL);\n \t}\n \n+\t/*\n+\t * The bitmap code actually stores the commit pointers for future\n+\t * reference, so we can't use this memory optimization there.\n+\t */\n+\tif (!write_bitmap_index)\n+\t\tfree_all_objects();\n+\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs(&revs);\n \tif (pack_loose_unreachable)\ndiff --git a/cache.h b/cache.h\nindex b90feb3802..605bab31de 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1872,6 +1872,8 @@ extern void *alloc_object_node(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\n \n+void alloc_clear_all(void);\n+\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/object.c b/object.c\nindex 9e6f9ff20b..6530d6fbde 100644\n--- a/object.c\n+++ b/object.c\n@@ -445,3 +445,12 @@ void clear_commit_marks_all(unsigned int flags)\n \t\t\tobj->flags &= ~flags;\n \t}\n }\n+\n+void free_all_objects(void)\n+{\n+\talloc_clear_all();\n+\tfree(obj_hash);\n+\tobj_hash = NULL;\n+\tobj_hash_size = 0;\n+\tnr_objs = 0;\n+}\ndiff --git a/object.h b/object.h\nindex 8ce294d6ec..3eb85215c2 100644\n--- a/object.h\n+++ b/object.h\n@@ -153,4 +153,6 @@ void clear_object_flags(unsigned flags);\n */\n extern void clear_commit_marks_all(unsigned int flags);\n \n+void free_all_objects(void);\n+\n #endif /* OBJECT_H */\n"},{"id":"342496","messageId":"20180322100145.GA21410@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8AF0FYysw48Y1CKdAHs_zWN8raYaGXrKQ9OmeKS-XujLQ@mail.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-22T10:01:45Z","receivedAt":"2018-03-22T10:01:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 22, 2018 at 09:23:42AM +0100, Duy Nguyen wrote:\n\n> > I wish you were right about the rarity, but it's unfortunately something\n> > I have seen multiple times in the wild (and why I spent time optimizing\n> > the many-packs case for pack-objects). Unfortunately I don't know how\n> > often it actually comes up, because in theory running \"git repack\"\n> > cleans it up without further ado. But after these patches, not so much.\n> \n> It's good to know this case is real and I can start to fix it\n> (assuming that the other concern about readability will not stop this\n> series).\n> \n> I think I'll try to fix this without involving repack. pack-objects\n> can produce multiple packs, so if we have more than 16k pack files, we\n> produce one new pack per 16k old ones.\n\nI suspect that's going to be hard given the structure of the code.\n\nCould we perhaps just bump to an auxiliary storage in that case? I.e.,\nallocate the final index number to mean \"look in this other table\", and\nthen have another table of uint32 indices?\n\nThat would mean we can behave as we did previously, but just use a\nlittle more memory in the uncommon >16k case.\n\n-Peff\n"},{"id":"342497","messageId":"CACsJy8CieaHxW-EE=3t4tdwAHoEg9+Bw3zpd+ZpROOc2k4Guxg@mail.gmail.com","threadId":"47943","inReplyTo":"20180322093212.GB28907@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-22T10:57:27Z","receivedAt":"2018-03-22T10:58:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:\n> That would still mean you could get into a broken state for serving\n> fetches, but you could at least get out of it by running \"git repack\".\n\nI was puzzled by this \"broken state\" statement. But you were right! I\nfocused on the repack case and forgot about fetch/clone case. I will\nprobably just drop this patch for now. Then maybe revisit this some\ntime in fiture when I find out how to deal with this nicely.\n-- \nDuy\n"},{"id":"342501","messageId":"20180322115247.GB22763@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8CieaHxW-EE=3t4tdwAHoEg9+Bw3zpd+ZpROOc2k4Guxg@mail.gmail.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-22T11:52:47Z","receivedAt":"2018-03-22T11:52:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote:\n\n> On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:\n> > That would still mean you could get into a broken state for serving\n> > fetches, but you could at least get out of it by running \"git repack\".\n> \n> I was puzzled by this \"broken state\" statement. But you were right! I\n> focused on the repack case and forgot about fetch/clone case. I will\n> probably just drop this patch for now. Then maybe revisit this some\n> time in fiture when I find out how to deal with this nicely.\n\nHere's a sketch of the \"separate array\" concept I mentioned before, in\ncase that helps. Not tested at all beyond compiling.\n\n---\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4406af640f..e4e308b453 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1090,7 +1090,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\toe_set_in_pack(entry, found_pack);\n+\t\toe_set_in_pack(&to_pack, entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \ndiff --git a/pack-objects.h b/pack-objects.h\nindex 9f8e450e19..b94b9232fa 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -7,6 +7,8 @@\n #define OE_Z_DELTA_BITS\t\t16\n #define OE_DELTA_SIZE_BITS\t31\n \n+#define OE_IN_PACK_EXTENDED ((1 << OE_IN_PACK_BITS) - 1)\n+\n /*\n * State flags for depth-first search used for analyzing delta cycles.\n *\n@@ -111,8 +113,13 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n-\tint in_pack_count;\n-\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n+\n+\tstruct packed_git **in_pack;\n+\tuint32_t in_pack_count;\n+\tsize_t in_pack_alloc;\n+\n+\tuint32_t *in_pack_extended;\n+\tsize_t in_pack_extended_alloc;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -174,17 +181,13 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n static inline unsigned int oe_add_pack(struct packing_data *pack,\n \t\t\t\t struct packed_git *p)\n {\n-\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n-\t\tdie(_(\"too many packs to handle in one go. \"\n-\t\t \"Please add .keep files to exclude\\n\"\n-\t\t \"some pack files and keep the number \"\n-\t\t \"of non-kept files below %d.\"),\n-\t\t 1 << OE_IN_PACK_BITS);\n \tif (p) {\n \t\tif (p->index > 0)\n \t\t\tdie(\"BUG: this packed is already indexed\");\n \t\tp->index = pack->in_pack_count;\n \t}\n+\tALLOC_GROW(pack->in_pack, pack->in_pack_count + 1,\n+\t\t pack->in_pack_alloc);\n \tpack->in_pack[pack->in_pack_count] = p;\n \treturn pack->in_pack_count++;\n }\n@@ -192,18 +195,28 @@ static inline unsigned int oe_add_pack(struct packing_data *pack,\n static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n \t\t\t\t\t const struct object_entry *e)\n {\n-\treturn pack->in_pack[e->in_pack_idx];\n-\n+\tuint32_t idx = e->in_pack_idx;\n+\tif (idx == OE_IN_PACK_EXTENDED)\n+\t\tidx = pack->in_pack_extended[e - pack->objects];\n+\treturn pack->in_pack[idx];\n }\n \n-static inline void oe_set_in_pack(struct object_entry *e,\n+static inline void oe_set_in_pack(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n \t\t\t\t struct packed_git *p)\n {\n \tif (p->index <= 0)\n \t\tdie(\"BUG: found_pack should be NULL \"\n \t\t \"instead of having non-positive index\");\n-\te->in_pack_idx = p->index;\n-\n+\telse if (p->index < OE_IN_PACK_EXTENDED)\n+\t\te->in_pack_idx = p->index;\n+\telse {\n+\t\tsize_t index = e - pack->objects;\n+\t\tALLOC_GROW(pack->in_pack_extended,\n+\t\t\t index, pack->in_pack_extended_alloc);\n+\t\tpack->in_pack_extended[index] = p->index;\n+\t\te->in_pack_idx = OE_IN_PACK_EXTENDED;\n+\t}\n }\n \n static inline struct object_entry *oe_delta(\n"},{"id":"342522","messageId":"CACsJy8AJQbUYsy2Bdsy5rFceq5wdTyYUfeW=GZ2XSnqLahCkKw@mail.gmail.com","threadId":"47943","inReplyTo":"20180322115247.GB22763@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-22T17:04:15Z","receivedAt":"2018-03-22T17:05:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 22, 2018 at 12:52 PM, Jeff King wrote:\n> On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote:\n>\n>> On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:\n>> > That would still mean you could get into a broken state for serving\n>> > fetches, but you could at least get out of it by running \"git repack\".\n>>\n>> I was puzzled by this \"broken state\" statement. But you were right! I\n>> focused on the repack case and forgot about fetch/clone case. I will\n>> probably just drop this patch for now. Then maybe revisit this some\n>> time in fiture when I find out how to deal with this nicely.\n>\n> Here's a sketch of the \"separate array\" concept I mentioned before, in\n> case that helps. Not tested at all beyond compiling.\n\nBrilliant! Sorry I couldn't read your suggestion carefully this morning.\n-- \nDuy\n"},{"id":"342550","messageId":"7a49135b-faad-9856-b757-e3ed4886720d@ramsayjones.plus.com","threadId":"47943","inReplyTo":"20180322093212.GB28907@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-03-23T01:28:12Z","receivedAt":"2018-03-23T01:28:21Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 22/03/18 09:32, Jeff King wrote:\n> On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:\n> \n>>> I hate to be a wet blanket, but am I the only one who is wondering\n>>> whether the tradeoffs is worth it? 8% memory reduction doesn't seem\n>>> mind-bogglingly good,\n>>\n>> AEvar measured RSS. If we count objects[] array alone, the saving is\n>> 40% (136 bytes per entry down to 80). Some is probably eaten up by\n>> mmap in rss.\n> \n> Measuring actual heap usage with massif, I get before/after peak heaps\n> of 1728 and 1346MB respectively when repacking linux.git. So that's ~22%\n> savings overall.\n> \n> Of the used heap after your patches:\n> \n> - ~40% of that is from packlist_alloc()\n> - ~17% goes to \"struct object\"\n> - ~10% for the object.c hash table to store all the \"struct object\"\n> - ~7% goes to the delta cache\n> - ~7% goes to the pack revindex (actually, there's a duplicate 7%\n> there, too; I think our peak is when we're sorting the revindex\n> and have to keep two copies in memory at once)\n\nwhich begs the question, how much slower would it be if we\nreplaced the radix-sort with an in-place sort (e.g. heapsort).\n\nI hacked up the patch below, just for fun. I don't have any\nlarge repos (or enough disk space) to do any meaningful perf\ntests, but I did at least compile it and it passes the test-suite.\n(That is no guarantee that I haven't introduced bugs, of course!)\n\n;-)\n\nATB,\nRamsay Jones\n\n-- >8 --\nSubject: [PATCH] pack-revindex: replace radix-sort with in-place heapsort\n\nSigned-off-by: Ramsay Jones \n---\n pack-revindex.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 60 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex ff5f62c03..16f17eac1 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -15,6 +15,7 @@\n * get the object sha1 from the main index.\n */\n \n+#ifdef DUMMY\n /*\n * This is a least-significant-digit radix sort.\n *\n@@ -112,6 +113,65 @@ static void sort_revindex(struct revindex_entry *entries, unsigned n, off_t max)\n #undef BUCKETS\n #undef DIGIT_SIZE\n }\n+#endif\n+\n+static inline void swap(struct revindex_entry *a, int i, int j)\n+{\n+\tstruct revindex_entry t;\n+\n+\tt = a[i];\n+\ta[i] = a[j];\n+\ta[j] = t;\n+}\n+\n+/*\n+ * assume that elements first .. last (array index first-1 .. last-1) obey\n+ * the partially ordered tree property, except possibly for the children of\n+ * the first element. push down the first element until the partially\n+ * ordered tree property is restored.\n+ */\n+static void push_down(struct revindex_entry *a, int first, int last)\n+{\n+\tint parent = first;\n+\tint last_node = last / 2;\n+\n+\twhile (parent <= last_node) {\n+\n+\t\tint left = 2 * parent;\n+\t\tint right = left + 1;\n+\t\tint biggest;\n+\n+\t\tif (right > last) /* parent only has one child */\n+\t\t\tbiggest = left;\n+\t\telse {\n+\t\t\tif (a[left-1].offset >= a[right-1].offset)\n+\t\t\t\tbiggest = left;\n+\t\t\telse\n+\t\t\t\tbiggest = right;\n+\n+\t\t}\n+\n+\t\tif (a[parent-1].offset >= a[biggest-1].offset)\n+\t\t\tbreak; /* partially ordered tree property, we're done */\n+\n+\t\t/* push parent down */\n+\t\tswap(a, parent-1, biggest-1);\n+\t\tparent = biggest;\n+\t}\n+}\n+\n+static void sort_revindex(struct revindex_entry *entries, unsigned n, off_t max)\n+{\n+\tint i;\n+\n+\tfor (i = n/2; i > 0; i--)\n+\t\tpush_down(entries, i, n);\n+\n+\tfor (i = n; i > 1; i--) {\n+\t\tswap(entries, 0, i-1);\n+\t\tpush_down(entries, 1, i-1);\n+\t}\n+}\n \n /*\n * Ordered list of offsets of objects in the pack.\n-- \n2.16.0\n\n"},{"id":"342556","messageId":"20180323024609.GA12229@sigill.intra.peff.net","threadId":"47943","inReplyTo":"7a49135b-faad-9856-b757-e3ed4886720d@ramsayjones.plus.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-23T02:46:10Z","receivedAt":"2018-03-23T02:46:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 23, 2018 at 01:28:12AM +0000, Ramsay Jones wrote:\n\n> > Of the used heap after your patches:\n> > \n> > - ~40% of that is from packlist_alloc()\n> > - ~17% goes to \"struct object\"\n> > - ~10% for the object.c hash table to store all the \"struct object\"\n> > - ~7% goes to the delta cache\n> > - ~7% goes to the pack revindex (actually, there's a duplicate 7%\n> > there, too; I think our peak is when we're sorting the revindex\n> > and have to keep two copies in memory at once)\n> \n> which begs the question, how much slower would it be if we\n> replaced the radix-sort with an in-place sort (e.g. heapsort).\n> \n> I hacked up the patch below, just for fun. I don't have any\n> large repos (or enough disk space) to do any meaningful perf\n> tests, but I did at least compile it and it passes the test-suite.\n> (That is no guarantee that I haven't introduced bugs, of course!)\n\nIt might have been easier to just revert 8b8dfd5132 (pack-revindex:\nradix-sort the revindex, 2013-07-11). It even includes some performance\nnumbers. :)\n\nIn short, no, I don't think we want to go back to a comparison-sort. The\nradix sort back then was around 4 times faster for linux.git. And that\nwas when there were half as many objects in the repository, so the radix\nsort should continue to improve as the repo size grows.\n\nThe absolute time savings aren't huge for something as bulky as a\nrepack, so it's less exciting in this context. But it's also not that\nmuch memory (7% of the peak here, but as I showed elsewhere, if we can\nstop holding all of the \"struct object\" memory once we're done with it,\nthen this revindex stuff doesn't even factor into the peak).\n\n-Peff\n"},{"id":"342561","messageId":"20180323055027.GA30227@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180323024609.GA12229@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-23T05:50:27Z","receivedAt":"2018-03-23T05:50:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote:\n\n> > which begs the question, how much slower would it be if we\n> > replaced the radix-sort with an in-place sort (e.g. heapsort).\n> > \n> > I hacked up the patch below, just for fun. I don't have any\n> > large repos (or enough disk space) to do any meaningful perf\n> > tests, but I did at least compile it and it passes the test-suite.\n> > (That is no guarantee that I haven't introduced bugs, of course!)\n> \n> It might have been easier to just revert 8b8dfd5132 (pack-revindex:\n> radix-sort the revindex, 2013-07-11). It even includes some performance\n> numbers. :)\n> \n> In short, no, I don't think we want to go back to a comparison-sort. The\n> radix sort back then was around 4 times faster for linux.git. And that\n> was when there were half as many objects in the repository, so the radix\n> sort should continue to improve as the repo size grows.\n\nI was curious whether my hand-waving there was true. It turns out that\nit is: the radix sort has stayed about the same speed but the comparison\nsort has gotten even slower. Here are best-of-five timings for \"git\ncat-file --batch-check='%(objectsize:disk)'\", which does very little\nbesides generate the rev-index:\n\n [current master, using radix sort]\n real\t0m0.104s\n user\t0m0.088s\n sys\t0m0.016s\n\n [reverting 8b8dfd5132, going back to qsort]\n real\t0m1.193s\n user\t0m1.176s\n sys\t0m0.016s\n\nSo it's now a factor of 11. Yikes.\n\nThat number does match some napkin math. The radix sort uses four 16-bit\nbuckets, but can quit when after two rounds (because none of the offsets\nis beyond 2^32). So it's essentially O(2n). Whereas the comparison sort\nis O(n log n), and with n around 6M, that puts log(n) right around 22.\n\nIt's possible that some other comparison-based sort might be a little\nmore efficient than qsort, but I don't think you'll be able to beat the\nalgorithmic speedup.\n\nThe revert of 8b8dfd5132 is below for reference (it needed a few\nconflict tweaks).\n\n-Peff\n\n---\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex ff5f62c033..c20aa9541b 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -15,102 +15,11 @@\n * get the object sha1 from the main index.\n */\n \n-/*\n- * This is a least-significant-digit radix sort.\n- *\n- * It sorts each of the \"n\" items in \"entries\" by its offset field. The \"max\"\n- * parameter must be at least as large as the largest offset in the array,\n- * and lets us quit the sort early.\n- */\n-static void sort_revindex(struct revindex_entry *entries, unsigned n, off_t max)\n+static int cmp_offset(const void *a_, const void *b_)\n {\n-\t/*\n-\t * We use a \"digit\" size of 16 bits. That keeps our memory\n-\t * usage reasonable, and we can generally (for a 4G or smaller\n-\t * packfile) quit after two rounds of radix-sorting.\n-\t */\n-#define DIGIT_SIZE (16)\n-#define BUCKETS (1 << DIGIT_SIZE)\n-\t/*\n-\t * We want to know the bucket that a[i] will go into when we are using\n-\t * the digit that is N bits from the (least significant) end.\n-\t */\n-#define BUCKET_FOR(a, i, bits) (((a)[(i)].offset >> (bits)) & (BUCKETS-1))\n-\n-\t/*\n-\t * We need O(n) temporary storage. Rather than do an extra copy of the\n-\t * partial results into \"entries\", we sort back and forth between the\n-\t * real array and temporary storage. In each iteration of the loop, we\n-\t * keep track of them with alias pointers, always sorting from \"from\"\n-\t * to \"to\".\n-\t */\n-\tstruct revindex_entry *tmp, *from, *to;\n-\tint bits;\n-\tunsigned *pos;\n-\n-\tALLOC_ARRAY(pos, BUCKETS);\n-\tALLOC_ARRAY(tmp, n);\n-\tfrom = entries;\n-\tto = tmp;\n-\n-\t/*\n-\t * If (max >> bits) is zero, then we know that the radix digit we are\n-\t * on (and any higher) will be zero for all entries, and our loop will\n-\t * be a no-op, as everybody lands in the same zero-th bucket.\n-\t */\n-\tfor (bits = 0; max >> bits; bits += DIGIT_SIZE) {\n-\t\tunsigned i;\n-\n-\t\tmemset(pos, 0, BUCKETS * sizeof(*pos));\n-\n-\t\t/*\n-\t\t * We want pos[i] to store the index of the last element that\n-\t\t * will go in bucket \"i\" (actually one past the last element).\n-\t\t * To do this, we first count the items that will go in each\n-\t\t * bucket, which gives us a relative offset from the last\n-\t\t * bucket. We can then cumulatively add the index from the\n-\t\t * previous bucket to get the true index.\n-\t\t */\n-\t\tfor (i = 0; i < n; i++)\n-\t\t\tpos[BUCKET_FOR(from, i, bits)]++;\n-\t\tfor (i = 1; i < BUCKETS; i++)\n-\t\t\tpos[i] += pos[i-1];\n-\n-\t\t/*\n-\t\t * Now we can drop the elements into their correct buckets (in\n-\t\t * our temporary array). We iterate the pos counter backwards\n-\t\t * to avoid using an extra index to count up. And since we are\n-\t\t * going backwards there, we must also go backwards through the\n-\t\t * array itself, to keep the sort stable.\n-\t\t *\n-\t\t * Note that we use an unsigned iterator to make sure we can\n-\t\t * handle 2^32-1 objects, even on a 32-bit system. But this\n-\t\t * means we cannot use the more obvious \"i >= 0\" loop condition\n-\t\t * for counting backwards, and must instead check for\n-\t\t * wrap-around with UINT_MAX.\n-\t\t */\n-\t\tfor (i = n - 1; i != UINT_MAX; i--)\n-\t\t\tto[--pos[BUCKET_FOR(from, i, bits)]] = from[i];\n-\n-\t\t/*\n-\t\t * Now \"to\" contains the most sorted list, so we swap \"from\" and\n-\t\t * \"to\" for the next iteration.\n-\t\t */\n-\t\tSWAP(from, to);\n-\t}\n-\n-\t/*\n-\t * If we ended with our data in the original array, great. If not,\n-\t * we have to move it back from the temporary storage.\n-\t */\n-\tif (from != entries)\n-\t\tCOPY_ARRAY(entries, tmp, n);\n-\tfree(tmp);\n-\tfree(pos);\n-\n-#undef BUCKET_FOR\n-#undef BUCKETS\n-#undef DIGIT_SIZE\n+\tconst struct revindex_entry *a = a_;\n+\tconst struct revindex_entry *b = b_;\n+\treturn (a->offset < b->offset) ? -1 : (a->offset > b->offset) ? 1 : 0;\n }\n \n /*\n@@ -152,7 +61,7 @@ static void create_pack_revindex(struct packed_git *p)\n \t */\n \tp->revindex[num_ent].offset = p->pack_size - 20;\n \tp->revindex[num_ent].nr = -1;\n-\tsort_revindex(p->revindex, num_ent, p->pack_size);\n+\tqsort(p->revindex, num_ent, sizeof(*p->revindex), cmp_offset);\n }\n \n void load_pack_revindex(struct packed_git *p)\n"},{"id":"342565","messageId":"CACsJy8BGykZed6uWg0cEvndDA3iHFaMCGv8tGrk5H0WfeGbgGA@mail.gmail.com","threadId":"47943","inReplyTo":"20180323024609.GA12229@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-23T07:05:47Z","receivedAt":"2018-03-23T07:06:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 23, 2018 at 3:46 AM, Jeff King wrote:\n> On Fri, Mar 23, 2018 at 01:28:12AM +0000, Ramsay Jones wrote:\n>\n>> > Of the used heap after your patches:\n>> >\n>> > - ~40% of that is from packlist_alloc()\n>> > - ~17% goes to \"struct object\"\n>> > - ~10% for the object.c hash table to store all the \"struct object\"\n>> > - ~7% goes to the delta cache\n>> > - ~7% goes to the pack revindex (actually, there's a duplicate 7%\n>> > there, too; I think our peak is when we're sorting the revindex\n>> > and have to keep two copies in memory at once)\n>>\n>> which begs the question, how much slower would it be if we\n>> replaced the radix-sort with an in-place sort (e.g. heapsort).\n>>\n>> I hacked up the patch below, just for fun. I don't have any\n>> large repos (or enough disk space) to do any meaningful perf\n>> tests, but I did at least compile it and it passes the test-suite.\n>> (That is no guarantee that I haven't introduced bugs, of course!)\n>\n> It might have been easier to just revert 8b8dfd5132 (pack-revindex:\n> radix-sort the revindex, 2013-07-11). It even includes some performance\n> numbers. :)\n>\n> In short, no, I don't think we want to go back to a comparison-sort. The\n> radix sort back then was around 4 times faster for linux.git. And that\n> was when there were half as many objects in the repository, so the radix\n> sort should continue to improve as the repo size grows.\n>\n> The absolute time savings aren't huge for something as bulky as a\n> repack, so it's less exciting in this context. But it's also not that\n> much memory (7% of the peak here, but as I showed elsewhere, if we can\n> stop holding all of the \"struct object\" memory once we're done with it,\n> then this revindex stuff doesn't even factor into the peak).\n\nWe probably could do something about revindex after rev-list memory is\nfreed up too. revindex is used in two places (i'm ignoring\npackfile.c): when we look for base objects in ofs-delta in\ncheck_objects() and when we write a reused object. The first case\ncan't be helped, it's why we need revindex. The second case though is\njust to get the compressed object size so we can copy the object.\n\nIf we cache the compressed size (like with uint32_t) then we can free\nup revindex after check_objects() is done. Even if we repack\neverything, this is still a very good saving (32 bits vs 128 bits per\nobject). The freed up memory could probably be used for delta cache.\nBut then if we hit a compressed object size larger than 4GB, revindex\nmust be recreated back, but we could detect this at check_objects\nphase and keep revindex alivie.\n-- \nDuy\n"},{"id":"342579","messageId":"478ddd61-4afe-44ae-c2b6-ecd3427a539f@ramsayjones.plus.com","threadId":"47943","inReplyTo":"20180323024609.GA12229@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-03-23T14:03:36Z","receivedAt":"2018-03-23T14:03:45Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 23/03/18 02:46, Jeff King wrote:\n> On Fri, Mar 23, 2018 at 01:28:12AM +0000, Ramsay Jones wrote:\n> \n>>> Of the used heap after your patches:\n>>>\n>>> - ~40% of that is from packlist_alloc()\n>>> - ~17% goes to \"struct object\"\n>>> - ~10% for the object.c hash table to store all the \"struct object\"\n>>> - ~7% goes to the delta cache\n>>> - ~7% goes to the pack revindex (actually, there's a duplicate 7%\n>>> there, too; I think our peak is when we're sorting the revindex\n>>> and have to keep two copies in memory at once)\n>>\n>> which begs the question, how much slower would it be if we\n>> replaced the radix-sort with an in-place sort (e.g. heapsort).\n>>\n>> I hacked up the patch below, just for fun. I don't have any\n>> large repos (or enough disk space) to do any meaningful perf\n>> tests, but I did at least compile it and it passes the test-suite.\n>> (That is no guarantee that I haven't introduced bugs, of course!)\n> \n> It might have been easier to just revert 8b8dfd5132 (pack-revindex:\n> radix-sort the revindex, 2013-07-11). It even includes some performance\n> numbers. :)\n\nBut not as much fun! :)\n\n> In short, no, I don't think we want to go back to a comparison-sort. The\n> radix sort back then was around 4 times faster for linux.git. And that\n> was when there were half as many objects in the repository, so the radix\n> sort should continue to improve as the repo size grows.\n\nYes, I expected radix-sort to be faster.\n\n> The absolute time savings aren't huge for something as bulky as a\n> repack, so it's less exciting in this context. But it's also not that\n> much memory (7% of the peak here, but as I showed elsewhere, if we can\n> stop holding all of the \"struct object\" memory once we're done with it,\n> then this revindex stuff doesn't even factor into the peak).\n\nI didn't see this post until afterwards. So, if it isn't even a\nfactor in the peak memory use, then it's clear this specific\nspace/time trade-off also isn't an issue! ;-)\n\nThanks.\n\nATB,\nRamsay Jones\n"},{"id":"342593","messageId":"4395a668-845e-a66f-2a14-f6a4474c3d4b@ramsayjones.plus.com","threadId":"47943","inReplyTo":"20180323055027.GA30227@sigill.intra.peff.net","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-03-23T16:01:50Z","receivedAt":"2018-03-23T16:01:57Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 23/03/18 05:50, Jeff King wrote:\n> On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote:\n[snip]\n> I was curious whether my hand-waving there was true. It turns out that\n> it is: the radix sort has stayed about the same speed but the comparison\n> sort has gotten even slower. Here are best-of-five timings for \"git\n> cat-file --batch-check='%(objectsize:disk)'\", which does very little\n> besides generate the rev-index:\n\nNot that it matters, but I assume this was something like:\n\n $ time (echo HEAD | git cat-file --batch-check=\"%(objectsize:disk)\")\n\n... and I suspect it was on the linux.git repo, yes?\n\nI used to have a copy of the linux repo on disk, but I had to\ndelete it a while ago to recover some disk space (no matter how\nbig disks get, they never seem big enough)!\n\nIf I do this on my biggest repo (ffmpeg), I get:\n\n $ cd ../ffmpeg/\n\n $ time (echo HEAD | git cat-file --batch-check=\"%(objectsize:disk)\")\n 227\n\n real\t0m0.037s\n user\t0m0.020s\n sys\t0m0.004s\n\n $ time (echo HEAD | ../git/git-cat-file --batch-check=\"%(objectsize:disk)\")\n 227\n\n real\t0m0.146s\n user\t0m0.112s\n sys\t0m0.012s\n\n $ \n\nWhere I'm using a version with my patch applied, rather than\nreverting commit 8b8dfd5132. A 395% slowdown is bad enough, but\nnot as bad as a factor of 11! I bet you have a much more modern\nsystem (with a fast SSD) than my old laptop. :-D\n\n> [current master, using radix sort]\n> real\t0m0.104s\n> user\t0m0.088s\n> sys\t0m0.016s\n> \n> [reverting 8b8dfd5132, going back to qsort]\n> real\t0m1.193s\n> user\t0m1.176s\n> sys\t0m0.016s\n> \n> So it's now a factor of 11. Yikes.\n\nThanks for looking into this, even if it was a wild\ngoose chase. :)\n\nATB,\nRamsay Jones\n\n"},{"id":"342727","messageId":"20180324063353.24722-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 01/13] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:41Z","receivedAt":"2018-03-24T06:34:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..c0a1f61aac 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,51 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 is\n+ * only valid after the object is written out and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size of the raw\n+ * data for an object in a pack is not stored anywhere but is computed\n+ * and made available when reverse .idx is made.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset) tuple contains the location of the\n+ * object in the source pack. in_pack_header_size allows quickly\n+ * skipping the header and going straight to the zlib stream.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created to\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342728","messageId":"20180324063353.24722-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 03/13] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:43Z","receivedAt":"2018-03-24T06:34:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 28 +++++++++++++++++-----------\n 2 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 647c01ea34..83f8154865 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b883d7aa10..8507e1b869 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -73,19 +88,10 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n+\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342729","messageId":"20180324063353.24722-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180318142526.9378-1-pclouds@gmail.com","subject":"[PATCH v7 00/13] nd/pack-objects-pack-struct updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:40Z","receivedAt":"2018-03-24T06:34:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Phew.. pack-objects is tough to crack. v7 changes\n\n- the 16k pack limit is removed thanks to Jeff suggestion. The limit\n for memory saving though is reduced down to 1k again.\n- only object size below 2G is cached (previously 4G) to avoid 1 << 32\n on 32 bits.\n- fix oe_size() retrieving wrong size (e.g. if entry->type is a delta\n then it must return delta size not canonical size)\n- fix race condition when oe_size() (and oe_delta_size()) accesses\n object database in parallel try_delta code.\n- add new GIT_TEST_ vars so that we use the whole test suite to\n exercise these hard-to-activate code paths.\n- I finally fixed the \"compressed\" typo!\n\nI'm not ignoring or forgetting other suggestions in this thread. the\nrev-list thing is still on my todo list (either do it externally or\nfree up mem as Jeff suggested). revindex as well. But those can be\ndone later.\n\nInterdiff\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex b8d936ccf5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -269,15 +269,6 @@ Unexpected missing object will raise an error.\n \tlocally created objects [without .promisor] and objects from the\n \tpromisor remote [with .promisor].) This is used with partial clone.\n \n-LIMITATIONS\n------------\n-\n-This command could only handle 16384 existing pack files at a time.\n-If you have more than this, you need to exclude some pack files with\n-\".keep\" file and --honor-pack-keep option, to combine 16k pack files\n-in one, then remove these .keep files and run pack-objects one more\n-time.\n-\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4406af640f..c774821930 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,8 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define SIZE(obj) oe_size(&to_pack, obj)\n+#define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n #define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n@@ -276,7 +278,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t oe_size_greater_than(entry, big_file_threshold) &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -386,7 +388,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n-\tunsigned long entry_size = oe_size(entry);\n+\tunsigned long entry_size = SIZE(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n@@ -1036,7 +1038,7 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (*found_pack) {\n \t\twant = want_found_object(exclude, *found_pack);\n \t\tif (want != -1)\n-\t\t\tgoto done;\n+\t\t\treturn want;\n \t}\n \n \tlist_for_each(pos, &packed_git_mru) {\n@@ -1059,16 +1061,11 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\tif (!exclude && want > 0)\n \t\t\t\tlist_move(&p->mru, &packed_git_mru);\n \t\t\tif (want != -1)\n-\t\t\t\tgoto done;\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n-\twant = 1;\n-done:\n-\tif (want && *found_pack && !(*found_pack)->index)\n-\t\toe_add_pack(&to_pack, *found_pack);\n-\n-\treturn want;\n+\treturn 1;\n }\n \n static void create_object_entry(const struct object_id *oid,\n@@ -1090,7 +1087,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\toe_set_in_pack(entry, found_pack);\n+\t\toe_set_in_pack(&to_pack, entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1415,7 +1412,7 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tunsigned long size;\n+\tunsigned long canonical_size;\n \n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n@@ -1427,6 +1424,7 @@ static void check_object(struct object_entry *entry)\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n \t\tenum object_type type;\n+\t\tunsigned long in_pack_size;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1436,14 +1434,13 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &size);\n+\t\t\t\t\t\t &in_pack_size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n \t\t\tdie(\"BUG: invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n-\t\toe_set_size(entry, size);\n \n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n@@ -1454,6 +1451,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tSET_SIZE(entry, in_pack_size);\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1510,8 +1508,9 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tSET_DELTA_SIZE(entry, oe_size(entry));\n+\t\t\tSET_DELTA_SIZE(entry, in_pack_size);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1519,18 +1518,18 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n-\t\t\tunsigned long size;\n+\t\t\toff_t delta_pos;\n \n-\t\t\tsize = get_size_from_delta(p, &w_curs,\n-\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\toe_set_size(entry, size);\n-\t\t\tif (oe_size_less_than(entry, 1))\n+\t\t\tdelta_pos = entry->in_pack_offset + entry->in_pack_header_size;\n+\t\t\tcanonical_size = get_size_from_delta(p, &w_curs, delta_pos);\n+\t\t\tif (canonical_size == 0)\n \t\t\t\tgoto give_up;\n+\t\t\tSET_SIZE(entry, canonical_size);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1544,15 +1543,18 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &size));\n-\t/*\n-\t * The error condition is checked in prepare_pack(). This is\n-\t * to permit a missing preferred base object to be ignored\n-\t * as a preferred base. Doing so can result in a larger\n-\t * pack file, but the transfer will still take place.\n-\t */\n-\tif (entry->type_valid)\n-\t\toe_set_size(entry, size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t &canonical_size));\n+\tif (entry->type_valid) {\n+\t\tSET_SIZE(entry, canonical_size);\n+\t} else {\n+\t\t/*\n+\t\t * Bad object type is checked in prepare_pack(). This is\n+\t\t * to permit a missing preferred base object to be ignored\n+\t\t * as a preferred base. Doing so can result in a larger\n+\t\t * pack file, but the transfer will still take place.\n+\t\t */\n+\t}\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1619,7 +1621,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n-\toe_set_size(entry, size);\n+\tSET_SIZE(entry, size);\n }\n \n /*\n@@ -1759,7 +1761,8 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (oe_size_greater_than(entry, big_file_threshold))\n+\t\tif (entry->type_valid &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1788,8 +1791,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n-\tunsigned long a_size = oe_size(a);\n-\tunsigned long b_size = oe_size(b);\n+\tunsigned long a_size = SIZE(a);\n+\tunsigned long b_size = SIZE(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1858,6 +1861,41 @@ static pthread_mutex_t progress_mutex;\n \n #endif\n \n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_window *w_curs;\n+\tunsigned char *buf;\n+\tenum object_type type;\n+\tunsigned long used, avail, size;\n+\n+\tif (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {\n+\t\tread_lock();\n+\t\tif (sha1_object_info(e->idx.oid.hash, &size) < 0)\n+\t\t\tdie(_(\"unable to get size of %s\"),\n+\t\t\t oid_to_hex(&e->idx.oid));\n+\t\tread_unlock();\n+\t\treturn size;\n+\t}\n+\n+\tp = oe_in_pack(pack, e);\n+\tif (!p)\n+\t\tdie(\"BUG: when e->type is a delta, it must belong to a pack\");\n+\n+\tread_lock();\n+\tw_curs = NULL;\n+\tbuf = use_pack(p, &w_curs, e->in_pack_offset, &avail);\n+\tused = unpack_object_header_buffer(buf, avail, &type, &size);\n+\tif (used == 0)\n+\t\tdie(_(\"unable to parse object header of %s\"),\n+\t\t oid_to_hex(&e->idx.oid));\n+\n+\tunuse_pack(&w_curs);\n+\tread_unlock();\n+\treturn size;\n+}\n+\n static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t unsigned max_depth, unsigned long *mem_usage)\n {\n@@ -1892,7 +1930,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = oe_size(trg_entry);\n+\ttrg_size = SIZE(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1904,7 +1942,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = oe_size(src_entry);\n+\tsrc_size = SIZE(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2026,7 +2064,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += oe_size(n->entry);\n+\t\tfreed_mem += SIZE(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2476,7 +2514,8 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (oe_size_less_than(entry, 50))\n+\t\tif (!entry->type_valid ||\n+\t\t oe_size_less_than(&to_pack, entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\n@@ -3235,9 +3274,6 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\t/* make sure IN_PACK(0) return NULL */\n-\toe_add_pack(&to_pack, NULL);\n-\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 9558d13834..59c6e40a02 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -2,6 +2,7 @@\n #include \"object.h\"\n #include \"pack.h\"\n #include \"pack-objects.h\"\n+#include \"packfile.h\"\n \n static uint32_t locate_object_entry_hash(struct packing_data *pdata,\n \t\t\t\t\t const unsigned char *sha1,\n@@ -86,15 +87,54 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \treturn &pdata->objects[pdata->index[i] - 1];\n }\n \n+static void prepare_in_pack_by_idx(struct packing_data *pdata)\n+{\n+\tstruct packed_git **mapping, *p;\n+\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n+\n+\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n+\t\t/*\n+\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n+\t\t * used instead\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tALLOC_ARRAY(mapping, nr);\n+\tmapping[cnt++] = NULL; /* zero index must be mapped to NULL */\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next, cnt++) {\n+\t\tif (cnt == nr) {\n+\t\t\tfree(mapping);\n+\t\t\treturn;\n+\t\t}\n+\t\tp->index = cnt;\n+\t\tmapping[cnt] = p;\n+\t}\n+\tpdata->in_pack_by_idx = mapping;\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos)\n {\n \tstruct object_entry *new_entry;\n \n+\tif (!pdata->nr_objects) {\n+\t\tprepare_in_pack_by_idx(pdata);\n+\t\tif (getenv(\"GIT_TEST_OE_SIZE_BITS\")) {\n+\t\t\tint bits = atoi(getenv(\"GIT_TEST_OE_SIZE_BITS\"));;\n+\t\t\tpdata->oe_size_limit = 1 << bits;\n+\t\t}\n+\t\tif (!pdata->oe_size_limit)\n+\t\t\tpdata->oe_size_limit = 1 << OE_SIZE_BITS;\n+\t}\n \tif (pdata->nr_objects >= pdata->nr_alloc) {\n \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\n+\n+\t\tif (!pdata->in_pack_by_idx)\n+\t\t\tREALLOC_ARRAY(pdata->in_pack, pdata->nr_alloc);\n \t}\n \n \tnew_entry = pdata->objects + pdata->nr_objects++;\n@@ -107,5 +147,8 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \telse\n \t\tpdata->index[index_pos] = pdata->nr_objects;\n \n+\tif (pdata->in_pack)\n+\t\tpdata->in_pack[pdata->nr_objects - 1] = NULL;\n+\n \treturn new_entry;\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex af40211105..9f19672602 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,8 +3,9 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n-#define OE_IN_PACK_BITS\t\t14\n+#define OE_IN_PACK_BITS\t\t10\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_SIZE_BITS\t\t31\n #define OE_DELTA_SIZE_BITS\t31\n \n /*\n@@ -35,7 +36,9 @@ enum dfs_state {\n *\n * \"size\" is the uncompressed object size. Compressed size of the raw\n * data for an object in a pack is not stored anywhere but is computed\n- * and made available when reverse .idx is made.\n+ * and made available when reverse .idx is made. Note that when an\n+ * delta is reused, \"size\" is the uncompressed _delta_ size, not the\n+ * canonical one after the delta has been applied.\n *\n * \"hash\" contains a path name hash which is used for sorting the\n * delta list and also during delta searching. Once prepare_pack()\n@@ -74,7 +77,8 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tuint32_t size_;\t/* object uncompressed size _if_ size_valid is true */\n+\tuint32_t size_:OE_SIZE_BITS;\n+\tunsigned size_valid:1;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n@@ -83,24 +87,23 @@ struct object_entry {\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n-\tunsigned size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n+\tunsigned no_try_delta:1;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n-\tunsigned no_try_delta:1;\n-\tunsigned char in_pack_header_size;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\tunsigned char in_pack_header_size;\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 80, bit_padding: 16 bits */\n+\t/* size: 80, bit_padding: 20 bits, holes: 1 bit */\n };\n \n struct packing_data {\n@@ -111,8 +114,17 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n-\tint in_pack_count;\n-\tstruct packed_git *in_pack[1 << OE_IN_PACK_BITS];\n+\n+\t/*\n+\t * Only one of these can be non-NULL and they have different\n+\t * sizes. if in_pack_by_idx is allocated, oe_in_pack() returns\n+\t * the pack of an object using in_pack_idx field. If not,\n+\t * in_pack[] array is used the same way as in_pack_pos[]\n+\t */\n+\tstruct packed_git **in_pack_by_idx;\n+\tstruct packed_git **in_pack;\n+\n+\tuintmax_t oe_size_limit;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -171,38 +183,27 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n-static inline unsigned int oe_add_pack(struct packing_data *pack,\n-\t\t\t\t struct packed_git *p)\n-{\n-\tif (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n-\t\tdie(_(\"too many packs to handle in one go. \"\n-\t\t \"Please add .keep files to exclude\\n\"\n-\t\t \"some pack files and keep the number \"\n-\t\t \"of non-kept files below %d.\"),\n-\t\t 1 << OE_IN_PACK_BITS);\n-\tif (p) {\n-\t\tif (p->index > 0)\n-\t\t\tdie(\"BUG: this packed is already indexed\");\n-\t\tp->index = pack->in_pack_count;\n-\t}\n-\tpack->in_pack[pack->in_pack_count] = p;\n-\treturn pack->in_pack_count++;\n-}\n-\n static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n \t\t\t\t\t const struct object_entry *e)\n {\n-\treturn pack->in_pack[e->in_pack_idx];\n+\tif (pack->in_pack_by_idx)\n+\t\treturn pack->in_pack_by_idx[e->in_pack_idx];\n+\telse\n+\t\treturn pack->in_pack[e - pack->objects];\n \n }\n \n-static inline void oe_set_in_pack(struct object_entry *e,\n+static inline void oe_set_in_pack(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n \t\t\t\t struct packed_git *p)\n {\n-\tif (p->index <= 0)\n-\t\tdie(\"BUG: found_pack should be NULL \"\n-\t\t \"instead of having non-positive index\");\n-\te->in_pack_idx = p->index;\n+\tif (pack->in_pack_by_idx) {\n+\t\tif (p->index <= 0)\n+\t\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t\t\t\t\"instead of having non-positive index\");\n+\t\t\te->in_pack_idx = p->index;\n+\t} else\n+\t\tpack->in_pack[e - pack->objects] = p;\n \n }\n \n@@ -263,56 +264,49 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n-static inline unsigned long oe_size(const struct object_entry *e)\n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e);\n+static inline unsigned long oe_size(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e)\n {\n-\tif (e->size_valid) {\n+\tif (e->size_valid)\n \t\treturn e->size_;\n-\t} else {\n-\t\tunsigned long size;\n \n-\t\tsha1_object_info(e->idx.oid.hash, &size);\n-\t\treturn size;\n-\t}\n+\treturn oe_get_size_slow(pack, e);\n }\n \n-static inline int oe_fits_in_32bits(unsigned long limit)\n+static inline int oe_size_less_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n {\n-\tuint32_t truncated_limit = (uint32_t)limit;\n-\n-\treturn limit == truncated_limit;\n-}\n-\n-static inline int oe_size_less_than(const struct object_entry *e,\n-\t\t\t\t unsigned long limit)\n-{\n-\tif (e->size_valid)\n-\t\treturn e->size_ < limit;\n-\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ < rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n \t\treturn 0;\n-\treturn oe_size(e) < limit;\n+\treturn oe_get_size_slow(pack, lhs) < rhs;\n }\n \n-static inline int oe_size_greater_than(const struct object_entry *e,\n-\t\t\t\t unsigned long limit)\n+static inline int oe_size_greater_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n {\n-\tif (e->size_valid)\n-\t\treturn e->size_ > limit;\n-\tif (oe_fits_in_32bits(limit)) /* limit < 2^32 <= size ? */\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ > rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n \t\treturn 1;\n-\treturn oe_size(e) > limit;\n+\treturn oe_get_size_slow(pack, lhs) > rhs;\n }\n \n-static inline void oe_set_size(struct object_entry *e,\n+static inline void oe_set_size(struct packing_data *pack,\n+\t\t\t struct object_entry *e,\n \t\t\t unsigned long size)\n {\n-\te->size_ = size;\n-\te->size_valid = e->size_ == size;\n-\n-\tif (!e->size_valid) {\n-\t\tunsigned long real_size;\n-\n-\t\tif (sha1_object_info(e->idx.oid.hash, &real_size) < 0 ||\n-\t\t size != real_size)\n+\tif (size < pack->oe_size_limit) {\n+\t\te->size_ = size;\n+\t\te->size_valid = 1;\n+\t} else {\n+\t\te->size_valid = 0;\n+\t\tif (oe_get_size_slow(pack, e) != size)\n \t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n \t}\n }\n@@ -322,7 +316,7 @@ static inline unsigned long oe_delta_size(struct packing_data *pack,\n {\n \tif (e->delta_size_valid)\n \t\treturn e->delta_size_;\n-\treturn oe_size(e);\n+\treturn oe_size(pack, e);\n }\n \n static inline void oe_set_delta_size(struct packing_data *pack,\n@@ -331,7 +325,7 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n {\n \te->delta_size_ = size;\n \te->delta_size_valid = e->delta_size_ == size;\n-\tif (!e->delta_size_valid && size != oe_size(e))\n+\tif (!e->delta_size_valid && size != oe_size(pack, e))\n \t\tdie(\"BUG: this can only happen in check_object() \"\n \t\t \"where delta size is the same as entry size\");\n }\ndiff --git a/t/README b/t/README\nindex 1a1361a806..da117ca734 100644\n--- a/t/README\n+++ b/t/README\n@@ -292,6 +292,26 @@ and know what setup is needed for it. Or when you want to run\n everything up to a certain test.\n \n \n+Running tests with special setups\n+---------------------------------\n+\n+The whole test suite could be run to test some special features\n+that cannot be easily covered by a few specific test cases. These\n+could be enabled by running the test suite with correct GIT_TEST_\n+environment set.\n+\n+GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n+\n+GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n+path where there are more than 1024 packs even if the actual number of\n+packs in repository is below this limit.\n+\n+GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n+code path where we do not cache objecct size in memory and read it\n+from existing packs on demand. This normally only happens when the\n+object size is over 2GB. This variable forces the code path on any\n+object larger than 2^ bytes.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 9c68b99251..5c076637ff 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -457,6 +457,11 @@ test_expect_success !PTHREADS,C_LOCALE_OUTPUT 'pack-objects --threads=N or pack.\n \tgrep -F \"no threads support, ignoring pack.threads\" err\n '\n \n+test_expect_success 'pack-objects in too-many-packs mode' '\n+\tGIT_TEST_FULL_IN_PACK_ARRAY=1 git repack -ad &&\n+\tgit fsck\n+'\n+\n #\n # WARNING!\n #\n\nNguyễn Thái Ngọc Duy (13):\n pack-objects: a bit of document about struct object_entry\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: shrink z_delta_size field in struct object_entry\n pack-objects: don't check size when the object is bad\n pack-objects: clarify the use of object_entry::size\n pack-objects: shrink size field in struct object_entry\n pack-objects: shrink delta_size field in struct object_entry\n pack-objects: reorder members to shrink struct object_entry\n\n Documentation/config.txt | 4 +-\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 4 +-\n builtin/pack-objects.c | 353 +++++++++++++++++++----------\n cache.h | 3 +\n object.h | 1 -\n pack-bitmap-write.c | 14 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.c | 43 ++++\n pack-objects.h | 306 ++++++++++++++++++++++---\n t/README | 10 +\n t/t5300-pack-object.sh | 5 +\n 13 files changed, 588 insertions(+), 165 deletions(-)\n\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342730","messageId":"20180324063353.24722-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 04/13] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:44Z","receivedAt":"2018-03-24T06:34:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Because of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 4 ++++\n pack-objects.h | 5 ++---\n 5 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 83f8154865..205e1f646c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth >= (1 << OE_DEPTH_BITS))\n+\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n+\t\t depth, (1 << OE_DEPTH_BITS) - 1);\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 8507e1b869..59407aae3c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -89,9 +90,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\tint depth;\n-\n+\tunsigned depth:OE_DEPTH_BITS;\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342731","messageId":"20180324063353.24722-14-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 13/13] pack-objects: reorder members to shrink struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:53Z","receivedAt":"2018-03-24T06:34:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Previous patches leave lots of holes and padding in this struct. This\npatch reorders the members and shrinks the struct down to 80 bytes\n(from 136 bytes, before any field shrinking is done) with 16 bits to\nspare (and a couple more in in_pack_header_size when we really run out\nof bits).\n\nThis is the last in a series of memory reduction patches (see\n\"pack-objects: a bit of document about struct object_entry\" for the\nfirst one).\n\nOverall they've reduced repack memory size on linux-2.6.git from\n3.747G to 3.424G, or by around 320M, a decrease of 8.5%. The runtime\nof repack has stayed the same throughout this series. Ævar's testing\non a big monorepo he has access to (bigger than linux-2.6.git) has\nshown a 7.9% reduction, so the overall expected improvement should be\nsomewhere around 8%.\n\nSee 87po42cwql.fsf@evledraar.gmail.com on-list\n(https://public-inbox.org/git/87po42cwql.fsf@evledraar.gmail.com/) for\nmore detailed numbers and a test script used to produce the numbers\ncited above.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 18 ++++++++++--------\n 1 file changed, 10 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 6d0345a82c..9f19672602 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -74,34 +74,36 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tuint32_t size_:OE_SIZE_BITS;\n-\tuint32_t size_valid:1;\n-\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n+\tuint32_t size_:OE_SIZE_BITS;\n+\tunsigned size_valid:1;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tuint32_t delta_size_valid:1;\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n+\tunsigned no_try_delta:1;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n-\tunsigned type_valid:1;\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n-\tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\tunsigned char in_pack_header_size;\n \tunsigned depth:OE_DEPTH_BITS;\n+\n+\t/* size: 80, bit_padding: 20 bits, holes: 1 bit */\n };\n \n struct packing_data {\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342732","messageId":"20180324063353.24722-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 11/13] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:51Z","receivedAt":"2018-03-24T06:34:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressed object is larger than 4GB\n(partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size\nis smaller than this limit.\n\nShrink size field down to 32 bits [1] and one overflow bit. If the\nsize is too large, we read it back from disk. As noted in the previous\npatch, we need to return the delta size instead of canonical size when\nthe to-be-reused object entry type is a delta instead of a canonical\none.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nAnother note about oe_get_size_slow(). This function MUST be thread\nsafe because SIZE() macro is used inside try_delta() which may run in\nparallel. Outside parallel code, no-contention locking should be dirt\ncheap (or insignificant compared to i/o access anyway). To exercise\nthis code, it's best to run the test suite with something like\n\n make test GIT_TEST_OE_SIZE_BITS=2\n\nwhich forces this code on all objects larger than 3 bytes.\n\n[1] it's actually already 32 bits on Windows\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 100 +++++++++++++++++++++++++++++++----------\n pack-objects.c | 9 +++-\n pack-objects.h | 53 +++++++++++++++++++++-\n t/README | 6 +++\n 4 files changed, 142 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f054ba9dfa..caeef086d3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,8 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define SIZE(obj) oe_size(&to_pack, obj)\n+#define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n@@ -274,7 +276,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +386,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = SIZE(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +409,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1407,6 +1410,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long canonical_size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1444,7 +1449,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->size = in_pack_size;\n+\t\t\tSET_SIZE(entry, in_pack_size);\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1501,9 +1506,9 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->size = in_pack_size; /* delta size */\n+\t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = in_pack_size;\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1519,9 +1524,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * object size from the delta header.\n \t\t\t */\n \t\t\tdelta_pos = entry->in_pack_offset + entry->in_pack_header_size;\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs, delta_pos);\n-\t\t\tif (entry->size == 0)\n+\t\t\tcanonical_size = get_size_from_delta(p, &w_curs, delta_pos);\n+\t\t\tif (canonical_size == 0)\n \t\t\t\tgoto give_up;\n+\t\t\tSET_SIZE(entry, canonical_size);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1535,13 +1541,18 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n-\t/*\n-\t * The error condition is checked in prepare_pack(). This is\n-\t * to permit a missing preferred base object to be ignored\n-\t * as a preferred base. Doing so can result in a larger\n-\t * pack file, but the transfer will still take place.\n-\t */\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t &canonical_size));\n+\tif (entry->type_valid) {\n+\t\tSET_SIZE(entry, canonical_size);\n+\t} else {\n+\t\t/*\n+\t\t * Bad object type is checked in prepare_pack(). This is\n+\t\t * to permit a missing preferred base object to be ignored\n+\t\t * as a preferred base. Doing so can result in a larger\n+\t\t * pack file, but the transfer will still take place.\n+\t\t */\n+\t}\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1592,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1605,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,10 +1615,11 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t\t &entry->size));\n+\t\t\t\t\t\t &size));\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n+\tSET_SIZE(entry, size);\n }\n \n /*\n@@ -1746,7 +1759,8 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (entry->type_valid && big_file_threshold < entry->size)\n+\t\tif (entry->type_valid &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1789,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n+\tunsigned long a_size = SIZE(a);\n+\tunsigned long b_size = SIZE(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1788,9 +1804,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1843,6 +1859,41 @@ static pthread_mutex_t progress_mutex;\n \n #endif\n \n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_window *w_curs;\n+\tunsigned char *buf;\n+\tenum object_type type;\n+\tunsigned long used, avail, size;\n+\n+\tif (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {\n+\t\tread_lock();\n+\t\tif (sha1_object_info(e->idx.oid.hash, &size) < 0)\n+\t\t\tdie(_(\"unable to get size of %s\"),\n+\t\t\t oid_to_hex(&e->idx.oid));\n+\t\tread_unlock();\n+\t\treturn size;\n+\t}\n+\n+\tp = oe_in_pack(pack, e);\n+\tif (!p)\n+\t\tdie(\"BUG: when e->type is a delta, it must belong to a pack\");\n+\n+\tread_lock();\n+\tw_curs = NULL;\n+\tbuf = use_pack(p, &w_curs, e->in_pack_offset, &avail);\n+\tused = unpack_object_header_buffer(buf, avail, &type, &size);\n+\tif (used == 0)\n+\t\tdie(_(\"unable to parse object header of %s\"),\n+\t\t oid_to_hex(&e->idx.oid));\n+\n+\tunuse_pack(&w_curs);\n+\tread_unlock();\n+\treturn size;\n+}\n+\n static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t unsigned max_depth, unsigned long *mem_usage)\n {\n@@ -1877,7 +1928,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = SIZE(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1940,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = SIZE(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2060,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += SIZE(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2510,8 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (!entry->type_valid || entry->size < 50)\n+\t\tif (!entry->type_valid ||\n+\t\t oe_size_less_than(&to_pack, entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 13f2b2bff2..59c6e40a02 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -120,8 +120,15 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n {\n \tstruct object_entry *new_entry;\n \n-\tif (!pdata->nr_objects)\n+\tif (!pdata->nr_objects) {\n \t\tprepare_in_pack_by_idx(pdata);\n+\t\tif (getenv(\"GIT_TEST_OE_SIZE_BITS\")) {\n+\t\t\tint bits = atoi(getenv(\"GIT_TEST_OE_SIZE_BITS\"));;\n+\t\t\tpdata->oe_size_limit = 1 << bits;\n+\t\t}\n+\t\tif (!pdata->oe_size_limit)\n+\t\t\tpdata->oe_size_limit = 1 << OE_SIZE_BITS;\n+\t}\n \tif (pdata->nr_objects >= pdata->nr_alloc) {\n \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex d23e17050c..8bb082f22f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,7 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t10\n #define OE_Z_DELTA_BITS\t\t16\n+#define OE_SIZE_BITS\t\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -72,7 +73,8 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\tuint32_t size_:OE_SIZE_BITS;\n+\tuint32_t size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n@@ -117,6 +119,8 @@ struct packing_data {\n \t */\n \tstruct packed_git **in_pack_by_idx;\n \tstruct packed_git **in_pack;\n+\n+\tuintmax_t oe_size_limit;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -256,4 +260,51 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e);\n+static inline unsigned long oe_size(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_;\n+\n+\treturn oe_get_size_slow(pack, e);\n+}\n+\n+static inline int oe_size_less_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n+{\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ < rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n+\t\treturn 0;\n+\treturn oe_get_size_slow(pack, lhs) < rhs;\n+}\n+\n+static inline int oe_size_greater_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n+{\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ > rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n+\t\treturn 1;\n+\treturn oe_get_size_slow(pack, lhs) > rhs;\n+}\n+\n+static inline void oe_set_size(struct packing_data *pack,\n+\t\t\t struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\tif (size < pack->oe_size_limit) {\n+\t\te->size_ = size;\n+\t\te->size_valid = 1;\n+\t} else {\n+\t\te->size_valid = 0;\n+\t\tif (oe_get_size_slow(pack, e) != size)\n+\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n+\t}\n+}\n+\n #endif\ndiff --git a/t/README b/t/README\nindex c6130ff16d..da117ca734 100644\n--- a/t/README\n+++ b/t/README\n@@ -306,6 +306,12 @@ GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n path where there are more than 1024 packs even if the actual number of\n packs in repository is below this limit.\n \n+GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n+code path where we do not cache objecct size in memory and read it\n+from existing packs on demand. This normally only happens when the\n+object size is over 2GB. This variable forces the code path on any\n+object larger than 2^ bytes.\n+\n Naming Tests\n ------------\n \n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342733","messageId":"20180324063353.24722-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 02/13] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:42Z","receivedAt":"2018-03-24T06:34:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"An extra field type_valid is added to carry the equivalent of OBJ_BAD\nin the original \"type\" field. in_pack_type always contains a valid\ntype so we only need 3 bits for it.\n\nA note about accepting OBJ_NONE as \"valid\" type. The function\nread_object_list_from_stdin() can pass this value [1] and it\neventually calls create_object_entry() where current code skip setting\n\"type\" field if the incoming type is zero. This does not have any bad\nside effects because \"type\" field should be memset()'d anyway.\n\nBut since we also need to set type_valid now, skipping oe_set_type()\nleaves type_valid zero/false, which will make oe_type() return\nOBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in\nprepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger\n\n fatal: unable to get type of object ...\n\nAccepting OBJ_NONE [2] does sound wrong, but this is how it is has\nbeen for a very long time and I haven't time to dig in further.\n\n[1] See 5c49c11686 (pack-objects: better check_object() performances -\n 2007-04-16)\n\n[2] 21666f1aae (convert object type handling from a string to a number\n - 2007-02-26)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 60 ++++++++++++++++++++++++------------------\n cache.h | 2 ++\n object.h | 1 -\n pack-bitmap-write.c | 6 ++---\n pack-objects.h | 20 ++++++++++++--\n 5 files changed, 58 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..647c01ea34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1066,8 +1067,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1428,9 +1433,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1484,7 +1489,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1493,7 +1498,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1516,7 +1521,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,16 +1576,18 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n-\t\t * And if that fails, the error will be recorded in entry->type\n+\t\t * And if that fails, the error will be recorded in oe_type(entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n+\t} else {\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1747,10 +1755,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1826,7 +1836,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2432,11 +2442,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2545,7 +2555,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..fd11f08940 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \n \t\tentry->in_pack_pos = i;\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -98,7 +98,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex c0a1f61aac..b883d7aa10 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -59,8 +59,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n+\tunsigned type_:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -123,4 +124,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342734","messageId":"20180324063353.24722-13-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 12/13] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:52Z","receivedAt":"2018-03-24T06:34:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n31 bits with one overflow bit.\n\nIf we find an existing delta larger than 2GB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nNote, since DELTA_SIZE() is used in try_delta() code, it must be\nthread-safe. Luckily oe_size() does guarantee this so we it is\nthread-safe.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 24 ++++++++++++++----------\n pack-objects.h | 23 ++++++++++++++++++++++-\n 2 files changed, 36 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex caeef086d3..c774821930 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -32,10 +32,12 @@\n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n #define SIZE(obj) oe_size(&to_pack, obj)\n #define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -142,7 +144,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -293,14 +295,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1508,7 +1510,7 @@ static void check_object(struct object_entry *entry)\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = in_pack_size;\n+\t\t\tSET_DELTA_SIZE(entry, in_pack_size);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1933,7 +1935,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -2004,10 +2006,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n+\t\treturn 0;\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -2022,7 +2026,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -2035,7 +2039,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2158,11 +2162,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 8bb082f22f..6d0345a82c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -6,6 +6,7 @@\n #define OE_IN_PACK_BITS\t\t10\n #define OE_Z_DELTA_BITS\t\t16\n #define OE_SIZE_BITS\t\t31\n+#define OE_DELTA_SIZE_BITS\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -83,7 +84,8 @@ struct object_entry {\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tuint32_t delta_size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -307,4 +309,23 @@ static inline void oe_set_size(struct packing_data *pack,\n \t}\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(pack, e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid = e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(pack, e))\n+\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342735","messageId":"20180324063353.24722-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:46Z","receivedAt":"2018-03-24T06:34:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index instead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 1k. Since we can't be\nsure people can never run into the situation where they have more than\n1k pack files. Provide a fall back route for it.\n\nIf we find out they have too many packs, the new in_pack_by_idx[]\narray (which has at most 1k elements) will not be used. Instead we\nallocate in_pack[] array that holds nr_objects elements. This is\nsimilar to how the optional in_pack_pos field is handled.\n\nThe new simple test is just to make sure the too-many-packs code path\nis at least executed. The true test is running\n\n make test GIT_TEST_FULL_IN_PACK_ARRAY=1\n\nto take advantage of other special case tests.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 26 +++++++++++++++-----------\n cache.h | 1 +\n pack-objects.c | 36 ++++++++++++++++++++++++++++++++++++\n pack-objects.h | 40 +++++++++++++++++++++++++++++++++++++++-\n t/README | 4 ++++\n t/t5300-pack-object.sh | 5 +++++\n 6 files changed, 100 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e1244918a5..b41610569e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -1074,7 +1076,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(&to_pack, entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1401,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1537,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1582,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1852,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 9558d13834..13f2b2bff2 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -2,6 +2,7 @@\n #include \"object.h\"\n #include \"pack.h\"\n #include \"pack-objects.h\"\n+#include \"packfile.h\"\n \n static uint32_t locate_object_entry_hash(struct packing_data *pdata,\n \t\t\t\t\t const unsigned char *sha1,\n@@ -86,15 +87,47 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \treturn &pdata->objects[pdata->index[i] - 1];\n }\n \n+static void prepare_in_pack_by_idx(struct packing_data *pdata)\n+{\n+\tstruct packed_git **mapping, *p;\n+\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n+\n+\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n+\t\t/*\n+\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n+\t\t * used instead\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tALLOC_ARRAY(mapping, nr);\n+\tmapping[cnt++] = NULL; /* zero index must be mapped to NULL */\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next, cnt++) {\n+\t\tif (cnt == nr) {\n+\t\t\tfree(mapping);\n+\t\t\treturn;\n+\t\t}\n+\t\tp->index = cnt;\n+\t\tmapping[cnt] = p;\n+\t}\n+\tpdata->in_pack_by_idx = mapping;\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos)\n {\n \tstruct object_entry *new_entry;\n \n+\tif (!pdata->nr_objects)\n+\t\tprepare_in_pack_by_idx(pdata);\n \tif (pdata->nr_objects >= pdata->nr_alloc) {\n \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\n+\n+\t\tif (!pdata->in_pack_by_idx)\n+\t\t\tREALLOC_ARRAY(pdata->in_pack, pdata->nr_alloc);\n \t}\n \n \tnew_entry = pdata->objects + pdata->nr_objects++;\n@@ -107,5 +140,8 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \telse\n \t\tpdata->index[index_pos] = pdata->nr_objects;\n \n+\tif (pdata->in_pack)\n+\t\tpdata->in_pack[pdata->nr_objects - 1] = NULL;\n+\n \treturn new_entry;\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 4a11653657..cae4f4fe01 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t10\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -18,6 +19,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 is\n@@ -65,7 +70,7 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -100,6 +105,15 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\n+\t/*\n+\t * Only one of these can be non-NULL and they have different\n+\t * sizes. if in_pack_by_idx is allocated, oe_in_pack() returns\n+\t * the pack of an object using in_pack_idx field. If not,\n+\t * in_pack[] array is used the same way as in_pack_pos[]\n+\t */\n+\tstruct packed_git **in_pack_by_idx;\n+\tstruct packed_git **in_pack;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -158,4 +172,28 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (pack->in_pack_by_idx)\n+\t\treturn pack->in_pack_by_idx[e->in_pack_idx];\n+\telse\n+\t\treturn pack->in_pack[e - pack->objects];\n+\n+}\n+\n+static inline void oe_set_in_pack(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (pack->in_pack_by_idx) {\n+\t\tif (p->index <= 0)\n+\t\t\tdie(\"BUG: found_pack should be NULL \"\n+\t\t\t\t\t\"instead of having non-positive index\");\n+\t\t\te->in_pack_idx = p->index;\n+\t} else\n+\t\tpack->in_pack[e - pack->objects] = p;\n+\n+}\n+\n #endif\ndiff --git a/t/README b/t/README\nindex 09eb2b9768..c6130ff16d 100644\n--- a/t/README\n+++ b/t/README\n@@ -302,6 +302,10 @@ environment set.\n \n GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n \n+GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n+path where there are more than 1024 packs even if the actual number of\n+packs in repository is below this limit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 9c68b99251..5c076637ff 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -457,6 +457,11 @@ test_expect_success !PTHREADS,C_LOCALE_OUTPUT 'pack-objects --threads=N or pack.\n \tgrep -F \"no threads support, ignoring pack.threads\" err\n '\n \n+test_expect_success 'pack-objects in too-many-packs mode' '\n+\tGIT_TEST_FULL_IN_PACK_ARRAY=1 git repack -ad &&\n+\tgit fsck\n+'\n+\n #\n # WARNING!\n #\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342736","messageId":"20180324063353.24722-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 05/13] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:45Z","receivedAt":"2018-03-24T06:34:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (it's not freed in the same way that objects[] is\nnot).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 16 +++++++++++++++-\n 5 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 205e1f646c..e1244918a5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -879,7 +879,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fd11f08940..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 59407aae3c..4a11653657 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -79,7 +79,6 @@ struct object_entry {\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -99,6 +98,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -144,4 +145,17 @@ static inline void oe_set_type(struct object_entry *e,\n \te->type_ = (unsigned)type;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342737","messageId":"20180324063353.24722-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 10/13] pack-objects: clarify the use of object_entry::size","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:50Z","receivedAt":"2018-03-24T06:34:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"While this field most of the time contains the canonical object size,\nthere is one case it does not: when we have found that the base object\nof the delta in question is also to be packed, we will very happily\nreuse the delta by copying it over instead of regenerating the new\ndelta.\n\n\"size\" in this case will record the delta size, not canonical object\nsize. Later on in write_reuse_object(), we reconstruct the delta\nheader and \"size\" is used for this purpose. When this happens, the\n\"type\" field contains a delta type instead of a canonical type.\nHighlight this in the code since it could be tricky to see.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 11 ++++++++---\n pack-objects.h | 4 +++-\n 2 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex da010f7d19..f054ba9dfa 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1417,6 +1417,7 @@ static void check_object(struct object_entry *entry)\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n \t\tenum object_type type;\n+\t\tunsigned long in_pack_size;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1426,7 +1427,7 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &in_pack_size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n@@ -1443,6 +1444,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tentry->size = in_pack_size;\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1499,6 +1501,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tentry->size = in_pack_size; /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n@@ -1508,13 +1511,15 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n+\t\t\toff_t delta_pos;\n+\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n+\t\t\tdelta_pos = entry->in_pack_offset + entry->in_pack_header_size;\n+\t\t\tentry->size = get_size_from_delta(p, &w_curs, delta_pos);\n \t\t\tif (entry->size == 0)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 415b961245..d23e17050c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -34,7 +34,9 @@ enum dfs_state {\n *\n * \"size\" is the uncompressed object size. Compressed size of the raw\n * data for an object in a pack is not stored anywhere but is computed\n- * and made available when reverse .idx is made.\n+ * and made available when reverse .idx is made. Note that when an\n+ * delta is reused, \"size\" is the uncompressed _delta_ size, not the\n+ * canonical one after the delta has been applied.\n *\n * \"hash\" contains a path name hash which is used for sorting the\n * delta list and also during delta searching. Once prepare_pack()\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342738","messageId":"20180324063353.24722-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 09/13] pack-objects: don't check size when the object is bad","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:49Z","receivedAt":"2018-03-24T06:34:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"sha1_object_info() in check_objects() may fail to locate an object in\nthe pack and return type OBJ_BAD. In that case, it will likely leave\nthe \"size\" field untouched. We delay error handling until later in\nprepare_pack() though. Until then, do not touch \"size\" field.\n\nThis field should contain the default value zero, but we can't say\nsha1_object_info() cannot damage it. This becomes more important later\nwhen the object size may have to be retrieved back from the\n(non-existing) pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b3e19815f1..da010f7d19 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1741,7 +1741,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (entry->type_valid && big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -2454,7 +2454,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (!entry->type_valid || entry->size < 50)\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342739","messageId":"20180324063353.24722-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 07/13] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:47Z","receivedAt":"2018-03-24T06:34:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These delta pointers always point to elements in the objects[] array\nin packing_data struct. We can only hold maximum 4G of those objects\nbecause the array size in nr_objects is uint32_t. We could use\nuint32_t indexes to address these elements instead of pointers. On\n64-bit architecture (8 bytes per pointer) this would save 4 bytes per\npointer.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 67 ++++++++++++++++++++++--\n 2 files changed, 124 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b41610569e..f7d3f6a1a8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -489,7 +495,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -541,12 +547,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -608,34 +614,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -646,7 +652,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -661,8 +667,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -672,11 +678,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1493,10 +1499,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1567,17 +1573,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1617,7 +1625,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1642,7 +1650,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1665,7 +1673,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1680,7 +1688,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1865,7 +1873,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1941,7 +1949,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1970,7 +1978,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1979,13 +1987,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2054,7 +2062,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2104,7 +2112,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2112,7 +2120,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2433,7 +2441,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex cae4f4fe01..433c194ffe 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -72,11 +72,11 @@ struct object_entry {\n \tunsigned long size;\t/* uncompressed size */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -196,4 +196,61 @@ static inline void oe_set_in_pack(struct packing_data *pack,\n \n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342740","messageId":"20180324063353.24722-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v7 08/13] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T06:33:48Z","receivedAt":"2018-03-24T06:34:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 16 bits, so you can only cache 65kb deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 22 ++++++++++++++++------\n pack-objects.h | 3 ++-\n 3 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f7d3f6a1a8..b3e19815f1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2100,12 +2100,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n+\t\t\t\tentry->z_delta_size = size;\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3084,6 +3091,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (depth >= (1 << OE_DEPTH_BITS))\n \t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n \t\t depth, (1 << OE_DEPTH_BITS) - 1);\n+\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n+\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n+\t\t (1 << OE_Z_DELTA_BITS) - 1);\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 433c194ffe..415b961245 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t10\n+#define OE_Z_DELTA_BITS\t\t16\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -79,7 +80,7 @@ struct object_entry {\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342741","messageId":"20180324064010.GA30422@sigill.intra.peff.net","threadId":"47943","inReplyTo":"4395a668-845e-a66f-2a14-f6a4474c3d4b@ramsayjones.plus.com","subject":"Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-24T06:40:10Z","receivedAt":"2018-03-24T06:40:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 23, 2018 at 04:01:50PM +0000, Ramsay Jones wrote:\n\n> Not that it matters, but I assume this was something like:\n> \n> $ time (echo HEAD | git cat-file --batch-check=\"%(objectsize:disk)\")\n> \n> ... and I suspect it was on the linux.git repo, yes?\n\nYes to both.\n\n> If I do this on my biggest repo (ffmpeg), I get:\n> \n> $ cd ../ffmpeg/\n> \n> $ time (echo HEAD | git cat-file --batch-check=\"%(objectsize:disk)\")\n> 227\n> \n> real\t0m0.037s\n> user\t0m0.020s\n> sys\t0m0.004s\n> \n> $ time (echo HEAD | ../git/git-cat-file --batch-check=\"%(objectsize:disk)\")\n> 227\n> \n> real\t0m0.146s\n> user\t0m0.112s\n> sys\t0m0.012s\n> \n> $ \n> \n> Where I'm using a version with my patch applied, rather than\n> reverting commit 8b8dfd5132. A 395% slowdown is bad enough, but\n> not as bad as a factor of 11! I bet you have a much more modern\n> system (with a fast SSD) than my old laptop. :-D\n\nYes, though it was all being run out of disk cache anyway. I also have a\nlot of RAM. :)\n\nThe ffmpeg repository only has ~550k objects. So that's log(19), and\nwe'd expect radix to be something like 8-9x faster than a comparison\nsort. But at some point the constants take over too (each O(n) round the\nradix sort actually has to look at the items twice, and of course there\nare negative cache effects when duplicating the array).\n\nSo your numbers match what I'd expect.\n\n> Thanks for looking into this, even if it was a wild\n> goose chase. :)\n\nNo problem. I think it's nice to sanity check my own hand-waving once in\na while. ;)\n\n-Peff\n"},{"id":"342744","messageId":"20180324072507.21059-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 1/7] t7700: have closing quote of a test at the beginning of line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:01Z","receivedAt":"2018-03-24T07:25:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The closing quote of a test body by convention is always at the start\nof line.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n t/t7700-repack.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 6061a04147..38247afbec 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -194,7 +194,7 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&\n \tgit repack -a -d &&\n \tgit cat-file -t $H1\n-\t'\n+'\n \n test_done\n \n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342745","messageId":"20180324072507.21059-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180316192745.19557-1-pclouds@gmail.com","subject":"[PATCH v4 0/7] nd/repack-keep-pack updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:00Z","receivedAt":"2018-03-24T07:25:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v4 is mostly refining tests and other minor fixes on v3\n\n- --keep-base-pack is renamed to --keep-largest-pack\n- \"Counting objects\" progress line is back\n- test and docs updates\n\nInterdiff\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 6b602f918f..cf862d3edf 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1562,7 +1562,8 @@ gc.bigPackThreshold::\n \tIf non-zero, all packs larger than this limit are kept when\n \t`git gc` is run. This is very similar to `--keep-base-pack`\n \texcept that all packs that meet the threshold are kept, not\n-\tjust the base pack. Defaults to zero.\n+\tjust the base pack. Defaults to zero. Common unit suffixes of\n+\t'k', 'm', or 'g' are supported.\n +\n Note that if the number of kept packs is more than gc.autoPackLimit,\n this configuration variable is ignored, all packs except the base pack\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 19b0d1741b..7549094900 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-base-pack]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -84,11 +84,10 @@ packs.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n---keep-base-pack::\n-\tAll packs except the base pack and those marked with a `.keep`\n-\tfiles are consolidated into a single pack. The largest pack is\n-\tconsidered the base pack. When this option is used,\n-\t`gc.bigPackThreshold` is ignored.\n+--keep-largest-pack::\n+\tAll packs except the largest pack and those marked with a\n+\t`.keep` files are consolidated into a single pack. When this\n+\toption is used, `gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 140c1bb7dd..d0b04e369e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -344,7 +344,7 @@ static int need_to_gc(void)\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n \t\t} else {\n-\t\t\tstruct packed_git * p = find_base_packs(&keep_pack, 0);\n+\t\t\tstruct packed_git *p = find_base_packs(&keep_pack, 0);\n \t\t\tuint64_t mem_have, mem_want;\n \n \t\t\tmem_have = total_ram();\n@@ -504,8 +504,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n-\t\tOPT_BOOL(0, \"keep-base-pack\", &keep_base_pack,\n-\t\t\t N_(\"repack all other packs except the base pack\")),\n+\t\tOPT_BOOL(0, \"keep-largest-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the largest pack\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d738627fc..1379b4cb92 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -54,7 +54,8 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep, ignore_packed_keep_in_core;\n+static int ignore_packed_keep_on_disk;\n+static int ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -983,14 +984,15 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep && !ignore_packed_keep_in_core &&\n+\tif (!ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n \tif (p->pack_local &&\n-\t ((ignore_packed_keep && p->pack_keep) ||\n+\t ((ignore_packed_keep_on_disk && p->pack_keep) ||\n \t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n@@ -1716,7 +1718,7 @@ static void get_object_details(void)\n \tstruct object_entry **sorted_by_offset;\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Getting object details\"),\n+\t\tprogress_state = start_progress(_(\"Counting objects\"),\n \t\t\t\t\t\tto_pack.nr_objects);\n \n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n@@ -2818,7 +2820,7 @@ static int pack_options_allow_reuse(void)\n {\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n-\t !ignore_packed_keep &&\n+\t !ignore_packed_keep_on_disk &&\n \t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n@@ -3059,7 +3061,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n-\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n+\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n \t\t\t\tN_(\"ignore this pack\")),\n@@ -3192,19 +3194,19 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tprepare_packed_git();\n \tadd_extra_kept_packs(&keep_pack_list);\n-\tif (ignore_packed_keep) {\n+\tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n-\t\t\tignore_packed_keep = 0;\n+\t\t\tignore_packed_keep_on_disk = 0;\n \t}\n \tif (local) {\n \t\t/*\n-\t\t * unlike ignore_packed_keep above, we do not want to\n-\t\t * unset \"local\" based on looking at packs, as it\n-\t\t * also covers non-local objects\n+\t\t * unlike ignore_packed_keep_on_disk above, we do not\n+\t\t * want to unset \"local\" based on looking at packs, as\n+\t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next) {\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 96ca70f9cc..100f287b97 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -50,6 +50,31 @@ test_expect_success 'gc is not aborted due to a stale symref' '\n \t)\n '\n \n+test_expect_success 'gc --keep-largest-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\t\ttest_commit three &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n+\t\ttest_commit five &&\n+\t\tgit repack -d &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 3 pack-list &&\n+\t\tgit gc --keep-largest-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_expect_success 'auto gc with too many loose objects does not attempt to create bitmaps' '\n \ttest_config gc.auto 3 &&\n \ttest_config gc.autodetach false &&\n@@ -123,28 +148,6 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_path_is_file .git/refs/heads/should-be-loose\n '\n \n-test_expect_success 'gc --keep-base-pack' '\n-\ttest_create_repo keep-pack &&\n-\t(\n-\t\tcd keep-pack &&\n-\t\tfor i in 10; do\n-\t\t\ttest_commit $i\n-\t\tdone &&\n-\t\tgit gc &&\n-\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n-\t\ttest_line_count = 1 pack-list &&\n-\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n-\t\tfor i in 10; do\n-\t\t\ttest_commit more-$i\n-\t\tdone &&\n-\t\tgit gc --keep-base-pack &&\n-\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n-\t\ttest_line_count = 2 pack-list &&\n-\t\ttest_path_is_file $BASE_PACK &&\n-\t\tgit fsck\n-\t)\n-'\n-\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 05ae0de3aa..6162e2a8e6 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -4,6 +4,12 @@ test_description='git repack works correctly'\n \n . ./test-lib.sh\n \n+commit_and_pack() {\n+\ttest_commit \"$@\" >/dev/null &&\n+\tSHA1=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack file1 &&\n \techo content2 > file2 &&\n@@ -200,22 +206,16 @@ test_expect_success 'repack --keep-pack' '\n \ttest_create_repo keep-pack &&\n \t(\n \t\tcd keep-pack &&\n-\t\ttest_commit one &&\n-\t\tgit repack -d &&\n-\t\ttest_commit two &&\n-\t\tgit repack -d &&\n-\t\ttest_commit three &&\n-\t\tgit repack -d &&\n-\t\ttest_commit four &&\n-\t\tgit repack -d &&\n-\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n-\t\ttest_line_count = 4 pack-list &&\n-\t\tKEEP1=$(head -n1 pack-list) &&\n-\t\tKEEP4=$(tail -n1 pack-list) &&\n-\t\tgit repack -a -d --keep-pack $KEEP1 --keep-pack $KEEP4 &&\n+\t\tP1=$(commit_and_pack 1) &&\n+\t\tP2=$(commit_and_pack 2) &&\n+\t\tP3=$(commit_and_pack 3) &&\n+\t\tP4=$(commit_and_pack 4) &&\n+\t\tls .git/objects/pack/*.pack >old-counts &&\n+\t\ttest_line_count = 4 old-counts &&\n+\t\tgit repack -a -d --keep-pack $P1 --keep-pack $P4 &&\n \t\tls .git/objects/pack/*.pack >new-counts &&\n-\t\tgrep -q $KEEP1 new-counts &&\n-\t\tgrep -q $KEEP4 new-counts &&\n+\t\tgrep -q $P1 new-counts &&\n+\t\tgrep -q $P4 new-counts &&\n \t\ttest_line_count = 3 new-counts &&\n \t\tgit fsck\n \t)\n\nNguyễn Thái Ngọc Duy (7):\n t7700: have closing quote of a test at the beginning of line\n repack: add --keep-pack option\n gc: add --keep-largest-pack option\n gc: add gc.bigPackThreshold config\n gc: handle a corner case in gc.bigPackThreshold\n gc --auto: exclude base pack if not enough mem to \"repack -ad\"\n pack-objects: show some progress when counting kept objects\n\n Documentation/config.txt | 12 +++\n Documentation/git-gc.txt | 19 +++-\n Documentation/git-pack-objects.txt | 9 +-\n Documentation/git-repack.txt | 9 +-\n builtin/gc.c | 167 +++++++++++++++++++++++++++--\n builtin/pack-objects.c | 84 +++++++++++----\n builtin/repack.c | 21 +++-\n cache.h | 1 +\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 32 ++++++\n t/t7700-repack.sh | 27 ++++-\n 13 files changed, 352 insertions(+), 36 deletions(-)\n\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342746","messageId":"20180324072507.21059-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 4/7] gc: add gc.bigPackThreshold config","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:04Z","receivedAt":"2018-03-24T07:25:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The --keep-largest-pack option is not very convenient to use because\nyou need to tell gc to do this explicitly (and probably on just a few\nlarge repos).\n\nAdd a config key that enables this mode when packs larger than a limit\nare found. Note that there's a slight behavior difference compared to\n--keep-largest-pack: all packs larger than the threshold are kept, not\njust the largest one.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 8 ++++++++\n Documentation/git-gc.txt | 6 ++++--\n builtin/gc.c | 26 ++++++++++++++++++++------\n 3 files changed, 32 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ce9102cea8..d63db3f12c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1558,6 +1558,14 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.bigPackThreshold::\n+\tIf non-zero, all packs larger than this limit are kept when\n+\t`git gc` is run. This is very similar to `--keep-base-pack`\n+\texcept that all packs that meet the threshold are kept, not\n+\tjust the base pack. Defaults to zero. Common unit suffixes of\n+\t'k', 'm', or 'g' are supported.\n+\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex bf81b8de30..0adf381b52 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -55,7 +55,8 @@ all loose objects are combined into a single pack using\n disables automatic packing of loose objects.\n +\n If the number of packs exceeds the value of `gc.autoPackLimit`,\n-then existing packs (except those marked with a `.keep` file)\n+then existing packs (except those marked with a `.keep` file\n+or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n automatic consolidation of packs.\n@@ -80,7 +81,8 @@ automatic consolidation of packs.\n \n --keep-largest-pack::\n \tAll packs except the largest pack and those marked with a\n-\t`.keep` files are consolidated into a single pack.\n+\t`.keep` files are consolidated into a single pack. When this\n+\toption is used, `gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9a09cf53b0..53a0500898 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -39,6 +39,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_pack_threshold;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -126,6 +127,8 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -164,7 +167,7 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs)\n+static void find_base_packs(struct string_list *packs, unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -173,7 +176,10 @@ static void find_base_packs(struct string_list *packs)\n \tfor (p = packed_git; p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n-\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\tif (limit) {\n+\t\t\tif (p->pack_size >= limit)\n+\t\t\t\tstring_list_append(packs, p->pack_name);\n+\t\t} else if (!base || base->pack_size < p->pack_size) {\n \t\t\tbase = p;\n \t\t}\n \t}\n@@ -245,9 +251,15 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option(NULL);\n-\telse if (too_many_loose_objects())\n+\tif (too_many_packs()) {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (big_pack_threshold)\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -462,7 +474,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t} else if (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342747","messageId":"20180324072507.21059-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 3/7] gc: add --keep-largest-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:03Z","receivedAt":"2018-03-24T07:25:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This adds a new repack mode that combines everything into a secondary\npack, leaving the largest pack alone.\n\nThis could help reduce memory pressure. On linux-2.6.git, valgrind\nmassif reports 1.6GB heap in \"pack all\" case, and 535MB in \"pack\nall except the base pack\" case. We save roughly 1GB memory by\nexcluding the base pack.\n\nThis should also lower I/O because we don't have to rewrite a giant\npack every time (e.g. for linux-2.6.git that's a 1.4GB pack file)..\n\nPS. The use of string_list here seems overkill, but we'll need it in\nthe next patch...\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 6 ++++-\n builtin/gc.c | 47 ++++++++++++++++++++++++++++++++++++----\n t/t6500-gc.sh | 25 +++++++++++++++++++++\n 3 files changed, 73 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..bf81b8de30 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -78,6 +78,10 @@ automatic consolidation of packs.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n+--keep-largest-pack::\n+\tAll packs except the largest pack and those marked with a\n+\t`.keep` files are consolidated into a single pack.\n+\n Configuration\n -------------\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..9a09cf53b0 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -164,6 +164,24 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n+static void find_base_packs(struct string_list *packs)\n+{\n+\tstruct packed_git *p, *base = NULL;\n+\n+\tprepare_packed_git();\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\t\tbase = p;\n+\t\t}\n+\t}\n+\n+\tif (base)\n+\t\tstring_list_append(packs, base->pack_name);\n+}\n+\n static int too_many_packs(void)\n {\n \tstruct packed_git *p;\n@@ -187,7 +205,13 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static int keep_one_pack(struct string_list_item *item, void *data)\n+{\n+\targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n+\treturn 0;\n+}\n+\n+static void add_repack_all_option(struct string_list *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -196,6 +220,9 @@ static void add_repack_all_option(void)\n \t\tif (prune_expire)\n \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n \t}\n+\n+\tif (keep_pack)\n+\t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n }\n \n static void add_repack_incremental_option(void)\n@@ -219,7 +246,7 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tadd_repack_all_option();\n+\t\tadd_repack_all_option(NULL);\n \telse if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -353,6 +380,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tconst char *name;\n \tpid_t pid;\n \tint daemonized = 0;\n+\tint keep_base_pack = -1;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -362,6 +390,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\")),\n \t\tOPT_BOOL(0, \"force\", &force, N_(\"force running gc even if there may be another gc running\")),\n+\t\tOPT_BOOL(0, \"keep-largest-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the largest pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -427,8 +457,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t */\n \t\t\tdaemonized = !daemonize();\n \t\t}\n-\t} else\n-\t\tadd_repack_all_option();\n+\t} else {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (keep_base_pack != -1) {\n+\t\t\tif (keep_base_pack)\n+\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t}\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t}\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 41b0be575d..a95969af2a 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -43,6 +43,31 @@ test_expect_success 'gc is not aborted due to a stale symref' '\n \t)\n '\n \n+test_expect_success 'gc --keep-largest-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\t\ttest_commit three &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n+\t\ttest_commit five &&\n+\t\tgit repack -d &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 3 pack-list &&\n+\t\tgit gc --keep-largest-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_expect_success 'auto gc with too many loose objects does not attempt to create bitmaps' '\n \ttest_config gc.auto 3 &&\n \ttest_config gc.autodetach false &&\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342748","messageId":"20180324072507.21059-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 6/7] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:06Z","receivedAt":"2018-03-24T07:25:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\ngiant base pack to avoid this problem is also known for a long time.\n\nRecent patches add an option to do just this, but it has to be either\nconfigured or activated manually. This patch lets `git gc --auto`\nactivate this mode automatically when it thinks `repack -ad` will use\na lot of memory and start affecting the system due to swapping or\nflushing OS cache.\n\ngc --auto decides to do this based on an estimation of pack-objects\nmemory usage, which is quite accurate at least for the heap part, and\nwhether that fits in half of system memory (the assumption here is for\ndesktop environment where there are many other applications running).\n\nThis mechanism only kicks in if gc.bigBasePackThreshold is not configured.\nIf it is, it is assumed that the user already knows what they want.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 9 +++-\n builtin/gc.c | 98 +++++++++++++++++++++++++++++++++++++++-\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n t/t6500-gc.sh | 7 +++\n 7 files changed, 119 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 0adf381b52..7549094900 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -58,8 +58,13 @@ If the number of packs exceeds the value of `gc.autoPackLimit`,\n then existing packs (except those marked with a `.keep` file\n or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n-'git repack'. Setting `gc.autoPackLimit` to 0 disables\n-automatic consolidation of packs.\n+'git repack'.\n+If the amount of memory is estimated not enough for `git repack` to\n+run smoothly and `gc.bigPackThreshold` is not set, the largest\n+pack will also be excluded (this is the equivalent of running `git gc`\n+with `--keep-base-pack`).\n+Setting `gc.autoPackLimit` to 0 disables automatic consolidation of\n+packs.\n \n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 74d3aaa270..d0b04e369e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,10 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -40,6 +44,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static unsigned long big_pack_threshold;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -128,6 +133,7 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -167,7 +173,8 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs, unsigned long limit)\n+static struct packed_git *find_base_packs(struct string_list *packs,\n+\t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -186,6 +193,8 @@ static void find_base_packs(struct string_list *packs, unsigned long limit)\n \n \tif (base)\n \t\tstring_list_append(packs, base->pack_name);\n+\n+\treturn base;\n }\n \n static int too_many_packs(void)\n@@ -211,6 +220,79 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n+static uint64_t total_ram(void)\n+{\n+#if defined(HAVE_SYSINFO)\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tsize_t length;\n+\n+\tmib[0] = CTL_HW;\n+# if defined(HW_MEMSIZE)\n+\tmib[1] = HW_MEMSIZE;\n+# else\n+\tmib[1] = HW_PHYSMEM;\n+# endif\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo.ullTotalPhys;\n+#endif\n+\treturn 0;\n+}\n+\n+static uint64_t estimate_repack_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t os_cache, heap;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\t/*\n+\t * First we have to scan through at least one pack.\n+\t * Assume enough room in OS file cache to keep the entire pack\n+\t * or we may accidentally evict data of other processes from\n+\t * the cache.\n+\t */\n+\tos_cache = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\theap = sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\theap += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\theap += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\theap += sizeof(struct object *) * nr_objects;\n+\t/* revindex is used also */\n+\theap += sizeof(struct revindex_entry) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\theap += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\theap += max_delta_cache_size;\n+\n+\treturn os_cache + heap;\n+}\n+\n static int keep_one_pack(struct string_list_item *item, void *data)\n {\n \targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n@@ -261,6 +343,20 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n+\t\t} else {\n+\t\t\tstruct packed_git *p = find_base_packs(&keep_pack, 0);\n+\t\t\tuint64_t mem_have, mem_want;\n+\n+\t\t\tmem_have = total_ram();\n+\t\t\tmem_want = estimate_repack_memory(p);\n+\n+\t\t\t/*\n+\t\t\t * Only allow 1/2 of memory for pack-objects, leave\n+\t\t\t * the rest for the OS and other processes in the\n+\t\t\t * system.\n+\t\t\t */\n+\t\t\tif (!mem_have || mem_want < mem_have / 2)\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7ec70131a9..8b2f8b72bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -80,7 +80,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 07e383257b..e373af48b8 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex a95969af2a..100f287b97 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -5,6 +5,13 @@ test_description='basic git gc tests\n \n . ./test-lib.sh\n \n+test_expect_success 'setup' '\n+\t# do not let the amount of physical memory affects gc\n+\t# behavior, make sure we always pack everything to one pack by\n+\t# default\n+\tgit config gc.bigPackThreshold 2g\n+'\n+\n test_expect_success 'gc empty repository' '\n \tgit gc\n '\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342749","messageId":"20180324072507.21059-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 7/7] pack-objects: show some progress when counting kept objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:07Z","receivedAt":"2018-03-24T07:25:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only show progress when there are new objects to be packed. But\nwhen --keep-pack is specified on the base pack, we will exclude most\nof objects. This makes 'pack-objects' stay silent for a long time\nwhile the counting phase is going.\n\nLet's show some progress whenever we visit an object instead. The old\n\"Counting objects\" is renamed to \"Enumerating objects\" and a new\nprogress \"Counting objects\" line is added.\n\nThis new \"Counting objects\" line should progress pretty quick when the\nsystem is beefy. But when the system is under pressure, the reading\nobject header done in this phase could be slow and showing progress is\nan improvement over staying silent in the current code.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8b2f8b72bf..1379b4cb92 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -44,7 +44,7 @@ static const char *pack_usage[] = {\n static struct packing_data to_pack;\n \n static struct pack_idx_entry **written_list;\n-static uint32_t nr_result, nr_written;\n+static uint32_t nr_result, nr_written, nr_seen;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n@@ -1096,6 +1096,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n \n@@ -1111,8 +1113,6 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tcreate_object_entry(oid, type, pack_name_hash(name),\n \t\t\t exclude, name && no_try_delta(name),\n \t\t\t index_pos, found_pack, found_offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1123,6 +1123,8 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n \n@@ -1130,8 +1132,6 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, index_pos, pack, offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1717,6 +1717,10 @@ static void get_object_details(void)\n \tuint32_t i;\n \tstruct object_entry **sorted_by_offset;\n \n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Counting objects\"),\n+\t\t\t\t\t\tto_pack.nr_objects);\n+\n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n \tfor (i = 0; i < to_pack.nr_objects; i++)\n \t\tsorted_by_offset[i] = to_pack.objects + i;\n@@ -1727,7 +1731,9 @@ static void get_object_details(void)\n \t\tcheck_object(entry);\n \t\tif (big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n+\t\tdisplay_progress(progress_state, i + 1);\n \t}\n+\tstop_progress(&progress_state);\n \n \t/*\n \t * This must happen in a second pass, since we rely on the delta\n@@ -3212,7 +3218,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n+\t\tprogress_state = start_progress(_(\"Enumerating objects\"), 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342750","messageId":"20180324072507.21059-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 5/7] gc: handle a corner case in gc.bigPackThreshold","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:05Z","receivedAt":"2018-03-24T07:25:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This config allows us to keep packs back if their size is larger\nthan a limit. But if this N >= gc.autoPackLimit, we may have a\nproblem. We are supposed to reduce the number of packs after a\nthreshold because it affects performance.\n\nWe could tell the user that they have incompatible gc.bigPackThreshold\nand gc.autoPackLimit, but it's kinda hard when 'git gc --auto' runs in\nbackground. Instead let's fall back to the next best stategy: try to\nreduce the number of packs anyway, but keep the base pack out. This\nreduces the number of packs to two and hopefully won't take up too\nmuch resources to repack (the assumption still is the base pack takes\nmost resources to handle).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 6 +++++-\n builtin/gc.c | 8 +++++++-\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d63db3f12c..cf862d3edf 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1564,7 +1564,11 @@ gc.bigPackThreshold::\n \texcept that all packs that meet the threshold are kept, not\n \tjust the base pack. Defaults to zero. Common unit suffixes of\n \t'k', 'm', or 'g' are supported.\n-\n++\n+Note that if the number of kept packs is more than gc.autoPackLimit,\n+this configuration variable is ignored, all packs except the base pack\n+will be repacked. After this the number of packs should go below\n+gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 53a0500898..74d3aaa270 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -254,8 +254,14 @@ static int need_to_gc(void)\n \tif (too_many_packs()) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold)\n+\t\tif (big_pack_threshold) {\n \t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n+\t\t\t\tbig_pack_threshold = 0;\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t}\n+\t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342751","messageId":"20180324072507.21059-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324072507.21059-1-pclouds@gmail.com","subject":"[PATCH v4 2/7] repack: add --keep-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T07:25:02Z","receivedAt":"2018-03-24T07:25:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We allow to keep existing packs by having companion .keep files. This\nis helpful when a pack is permanently kept. In the next patch, git-gc\njust wants to keep a pack temporarily, for one pack-objects\nrun. git-gc can use --keep-pack for this use case.\n\nA note about why the pack_keep field cannot be reused and\npack_keep_in_core has to be added. This is about the case when\n--keep-pack is specified together with either --keep-unreachable or\n--unpack-unreachable, but --honor-pack-keep is NOT specified.\n\nIn this case, we want to exclude objects from the packs specified on\ncommand line, not from ones with .keep files. If only one bit flag is\nused, we have to clear pack_keep on pack files with the .keep file.\n\nBut we can't make any assumption about unreachable objects in .keep\npacks. If \"pack_keep\" field is false for .keep packs, we could\npotentially pull lots of unreachable objects into the new pack, or\nunpack them loose. The safer approach is ignore all packs with either\n.keep file or --keep-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++-\n Documentation/git-repack.txt | 9 ++++-\n builtin/pack-objects.c | 64 ++++++++++++++++++++++++------\n builtin/repack.c | 21 ++++++++--\n cache.h | 1 +\n t/t7700-repack.sh | 25 ++++++++++++\n 6 files changed, 111 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..403524652a 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]]\n+\t[--revs [--unpacked | --all]] [--keep-pack=]\n \t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n@@ -126,6 +126,13 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n+--keep-pack=::\n+\tThis flag causes an object already in the given pack to be\n+\tignored, even if it would have otherwise been\n+\tpacked. `` is the the pack file name without\n+\tleading directory (e.g. `pack-123.pack`). The option could be\n+\tspecified multiple times to keep multiple packs.\n+\n --incremental::\n \tThis flag causes an object already in a pack to be ignored\n \teven if it would have otherwise been packed.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..ce497d9d12 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=] [--keep-pack=]\n \n DESCRIPTION\n -----------\n@@ -133,6 +133,13 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--keep-pack=::\n+\tExclude the given pack from repacking. This is the equivalent\n+\tof having `.keep` file on the pack. `` is the the\n+\tpack file name without leading directory (e.g. `pack-123.pack`).\n+\tThe option could be specified multiple times to keep multiple\n+\tpacks.\n+\n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\n \tobjects older than ``. This can be used to optimize out\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a197926eaa..7ec70131a9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -28,6 +28,7 @@\n #include \"argv-array.h\"\n #include \"list.h\"\n #include \"packfile.h\"\n+#include \"dir.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -53,7 +54,8 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep;\n+static int ignore_packed_keep_on_disk;\n+static int ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -982,13 +984,16 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep &&\n+\tif (!ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n-\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\tif (p->pack_local &&\n+\t ((ignore_packed_keep_on_disk && p->pack_keep) ||\n+\t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n \t/* we don't know yet; keep looking for more packs */\n@@ -2677,7 +2682,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \t\tstruct object_id oid;\n \t\tstruct object *o;\n \n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n@@ -2739,7 +2744,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \tp = (last_found != (void *)1) ? last_found : packed_git;\n \n \twhile (p) {\n-\t\tif ((!p->pack_local || p->pack_keep) &&\n+\t\tif ((!p->pack_local || p->pack_keep ||\n+\t\t\t\tp->pack_keep_in_core) &&\n \t\t\tfind_pack_entry_one(oid->hash, p)) {\n \t\t\tlast_found = p;\n \t\t\treturn 1;\n@@ -2782,7 +2788,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tstruct object_id oid;\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n \t\tif (open_pack_index(p))\n@@ -2808,7 +2814,8 @@ static int pack_options_allow_reuse(void)\n {\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n-\t !ignore_packed_keep &&\n+\t !ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n }\n@@ -2917,6 +2924,33 @@ static void get_object_list(int ac, const char **av)\n \toid_array_clear(&recent_objects);\n }\n \n+static void add_extra_kept_packs(const struct string_list *names)\n+{\n+\tstruct packed_git *p;\n+\n+\tif (!names->nr)\n+\t\treturn;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tconst char *name = basename(p->pack_name);\n+\t\tint i;\n+\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < names->nr; i++)\n+\t\t\tif (!fspathcmp(name, names->items[i].string))\n+\t\t\t\tbreak;\n+\n+\t\tif (i < names->nr) {\n+\t\t\tp->pack_keep_in_core = 1;\n+\t\t\tignore_packed_keep_in_core = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+}\n+\n static int option_parse_index_version(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -2956,6 +2990,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t N_(\"do not show progress meter\"), 0),\n@@ -3020,8 +3055,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n-\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n+\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n \t\t\t N_(\"pack compression level\")),\n \t\tOPT_SET_INT(0, \"keep-true-parents\", &grafts_replace_parents,\n@@ -3150,19 +3187,20 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tprogress = 2;\n \n \tprepare_packed_git();\n-\tif (ignore_packed_keep) {\n+\tadd_extra_kept_packs(&keep_pack_list);\n+\tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n-\t\t\tignore_packed_keep = 0;\n+\t\t\tignore_packed_keep_on_disk = 0;\n \t}\n \tif (local) {\n \t\t/*\n-\t\t * unlike ignore_packed_keep above, we do not want to\n-\t\t * unset \"local\" based on looking at packs, as it\n-\t\t * also covers non-local objects\n+\t\t * unlike ignore_packed_keep_on_disk above, we do not\n+\t\t * want to unset \"local\" based on looking at packs, as\n+\t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n \t\tfor (p = packed_git; p; p = p->next) {\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..6c636e159e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n * have a corresponding .keep or .promisor file. These packs are not to\n * be kept if we are going to pack everything into one file.\n */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list)\n+static void get_non_kept_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t\tconst struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \twhile ((e = readdir(dir)) != NULL) {\n \t\tsize_t len;\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < extra_keep->nr; i++)\n+\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n+\t\t\t\tbreak;\n+\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n+\t\t\tcontinue;\n+\n \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n \t\t\tcontinue;\n \n@@ -148,7 +157,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tint ext, ret, failed;\n+\tint i, ext, ret, failed;\n \tFILE *out;\n \n \t/* variables to be filled by option parsing */\n@@ -160,6 +169,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *depth = NULL;\n \tconst char *threads = NULL;\n \tconst char *max_pack_size = NULL;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_reuse_delta = 0, no_reuse_object = 0;\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n@@ -200,6 +210,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"maximum size of each packfile\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -230,6 +242,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--keep-true-parents\");\n \tif (!pack_kept_objects)\n \t\targv_array_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list.nr; i++)\n+\t\targv_array_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t\t keep_pack_list.items[i].string);\n \targv_array_push(&cmd.args, \"--non-empty\");\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n@@ -254,7 +269,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&cmd.args, \"--write-bitmap-index\");\n \n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs);\n+\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n \t\t\tif (unpack_unreachable) {\ndiff --git a/cache.h b/cache.h\nindex d06932ed0b..ba168e3571 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint pack_fd;\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n+\t\t pack_keep_in_core:1,\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1;\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 38247afbec..6162e2a8e6 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -4,6 +4,12 @@ test_description='git repack works correctly'\n \n . ./test-lib.sh\n \n+commit_and_pack() {\n+\ttest_commit \"$@\" >/dev/null &&\n+\tSHA1=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack file1 &&\n \techo content2 > file2 &&\n@@ -196,5 +202,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit cat-file -t $H1\n '\n \n+test_expect_success 'repack --keep-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tP1=$(commit_and_pack 1) &&\n+\t\tP2=$(commit_and_pack 2) &&\n+\t\tP3=$(commit_and_pack 3) &&\n+\t\tP4=$(commit_and_pack 4) &&\n+\t\tls .git/objects/pack/*.pack >old-counts &&\n+\t\ttest_line_count = 4 old-counts &&\n+\t\tgit repack -a -d --keep-pack $P1 --keep-pack $P4 &&\n+\t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\tgrep -q $P1 new-counts &&\n+\t\tgrep -q $P4 new-counts &&\n+\t\ttest_line_count = 3 new-counts &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n \n-- \n2.17.0.rc0.348.gd5a49e0b6f\n\n"},{"id":"342810","messageId":"87efk9yfm2.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-7-pclouds@gmail.com","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-24T09:42:13Z","receivedAt":"2018-03-24T09:42:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 24 2018, Nguyễn Thái Ngọc Duy wrote:\n\n> +\tif (pack->in_pack_by_idx) {\n> +\t\tif (p->index <= 0)\n> +\t\t\tdie(\"BUG: found_pack should be NULL \"\n> +\t\t\t\t\t\"instead of having non-positive index\");\n> +\t\t\te->in_pack_idx = p->index;\n> +\t} else\n\nThe indentation after the die() here is wrong. GCC complaining about it:\n \n ./pack-objects.h: In function ‘oe_set_in_pack’:\n ./pack-objects.h:203:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]\n if (p->index <= 0)\n ^~\n ./pack-objects.h:206:4: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’\n e->in_pack_idx = p->index;\n ^\n"},{"id":"342815","messageId":"87d0zty8mf.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-7-pclouds@gmail.com","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-24T12:13:12Z","receivedAt":"2018-03-24T12:13:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 24 2018, Nguyễn Thái Ngọc Duy wrote:\n\n> Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\n> pack. Use an index instead since the number of packs should be\n> relatively small.\n>\n> This limits the number of packs we can handle to 1k. Since we can't be\n> sure people can never run into the situation where they have more than\n> 1k pack files. Provide a fall back route for it.\n>\n> If we find out they have too many packs, the new in_pack_by_idx[]\n> array (which has at most 1k elements) will not be used. Instead we\n> allocate in_pack[] array that holds nr_objects elements. This is\n> similar to how the optional in_pack_pos field is handled.\n>\n> The new simple test is just to make sure the too-many-packs code path\n> is at least executed. The true test is running\n>\n> make test GIT_TEST_FULL_IN_PACK_ARRAY=1\n\nAside from the tiny nit in 87efk9yfm2.fsf@evledraar.gmail.com this looks\ngood to me.\n\nI've tested this with the same method noted in\n87vadpxv27.fsf@evledraar.gmail.com against the version before it on\nsimilar test data, and got:\n\n * Reduction in user time by 0.42%\n * Reduction in system time by 3.17%\n * Reduction in RSS by 0.003209%\n * Reduction in page faults by 0% & 0.006539% (time(1) reports two different numbers)\n * Reduction in the I of I/O by 99.504132% (note: from 4840 bytes to 24 bytes, so some fluke...)\n * Reduction in the O of I/O by 0%\n\nI.e. there's no notable change at all, but I thought it would be useful\nto re-run this for context.\n"},{"id":"342817","messageId":"CACsJy8B=TYTOu=4i0+0bqHRMDfJCQW5JA=3-g+v=D6pi2dbULw@mail.gmail.com","threadId":"47943","inReplyTo":"87efk9yfm2.fsf@evledraar.gmail.com","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-24T12:26:10Z","receivedAt":"2018-03-24T12:26:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 24, 2018 at 10:42 AM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Sat, Mar 24 2018, Nguyễn Thái Ngọc Duy wrote:\n>\n>> + if (pack->in_pack_by_idx) {\n>> + if (p->index <= 0)\n>> + die(\"BUG: found_pack should be NULL \"\n>> + \"instead of having non-positive index\");\n>> + e->in_pack_idx = p->index;\n>> + } else\n>\n> The indentation after the die() here is wrong. GCC complaining about it:\n>\n> ./pack-objects.h: In function ‘oe_set_in_pack’:\n> ./pack-objects.h:203:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]\n> if (p->index <= 0)\n> ^~\n> ./pack-objects.h:206:4: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’\n> e->in_pack_idx = p->index;\n> ^\n\nThanks. My gcc reported the same thing but only when not used with ccache, hm...\n-- \nDuy\n"},{"id":"343013","messageId":"20180326151317.GA24425@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"Re: [PATCH v7 00/13] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-26T15:13:17Z","receivedAt":"2018-03-26T15:13:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:40AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> +unsigned long oe_get_size_slow(struct packing_data *pack,\n> +\t\t\t const struct object_entry *e)\n> +{\n> +\tstruct packed_git *p;\n> +\tstruct pack_window *w_curs;\n> +\tunsigned char *buf;\n> +\tenum object_type type;\n> +\tunsigned long used, avail, size;\n> +\n> +\tif (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {\n> +\t\tread_lock();\n> +\t\tif (sha1_object_info(e->idx.oid.hash, &size) < 0)\n> +\t\t\tdie(_(\"unable to get size of %s\"),\n> +\t\t\t oid_to_hex(&e->idx.oid));\n> +\t\tread_unlock();\n> +\t\treturn size;\n> +\t}\n> +\n> +\tp = oe_in_pack(pack, e);\n> +\tif (!p)\n> +\t\tdie(\"BUG: when e->type is a delta, it must belong to a pack\");\n> +\n> +\tread_lock();\n> +\tw_curs = NULL;\n> +\tbuf = use_pack(p, &w_curs, e->in_pack_offset, &avail);\n> +\tused = unpack_object_header_buffer(buf, avail, &type, &size);\n> +\tif (used == 0)\n> +\t\tdie(_(\"unable to parse object header of %s\"),\n> +\t\t oid_to_hex(&e->idx.oid));\n> +\n> +\tunuse_pack(&w_curs);\n> +\tread_unlock();\n> +\treturn size;\n> +}\n\nIt took me a while to figure out why this treated deltas and non-deltas\ndifferently. At first I thought it was an optimization (since we can\nfind non-delta sizes quickly by looking at the headers). But I think\nit's just that you want to know the size of the actual _delta_, not the\nreconstructed object. And there's no way to ask sha1_object_info() for\nthat.\n\nPerhaps the _extended version of that function should learn an\nOBJECT_INFO_NO_DEREF flag or something to tell it return the true delta\ntype and size. Then this whole function could just become a single call.\n\nBut short of that, it's probably worth a comment explaining what's going\non.\n\n> +static void prepare_in_pack_by_idx(struct packing_data *pdata)\n> +{\n> +\tstruct packed_git **mapping, *p;\n> +\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n> +\n> +\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n> +\t\t/*\n> +\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n> +\t\t * used instead\n> +\t\t */\n> +\t\treturn;\n> +\t}\n\nMinor nit, but can we use git_env_bool() here? It's just as easy, and\nit's less surprising in some corner cases.\n\n> struct object_entry *packlist_alloc(struct packing_data *pdata,\n> \t\t\t\t const unsigned char *sha1,\n> \t\t\t\t uint32_t index_pos)\n> {\n> \tstruct object_entry *new_entry;\n> \n> +\tif (!pdata->nr_objects) {\n> +\t\tprepare_in_pack_by_idx(pdata);\n> +\t\tif (getenv(\"GIT_TEST_OE_SIZE_BITS\")) {\n> +\t\t\tint bits = atoi(getenv(\"GIT_TEST_OE_SIZE_BITS\"));;\n> +\t\t\tpdata->oe_size_limit = 1 << bits;\n> +\t\t}\n> +\t\tif (!pdata->oe_size_limit)\n> +\t\t\tpdata->oe_size_limit = 1 << OE_SIZE_BITS;\n> +\t}\n\nDitto here; I think this could just be:\n\n pdata->oe_size_limit = git_env_ulong(\"GIT_TEST_OE_SIZE_BITS\",\n (1 << OE_SIZE_BITS));\n\n> \tif (pdata->nr_objects >= pdata->nr_alloc) {\n> \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n> \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\n> +\n> +\t\tif (!pdata->in_pack_by_idx)\n> +\t\t\tREALLOC_ARRAY(pdata->in_pack, pdata->nr_alloc);\n> \t}\n\nI was going to complain that we don't use ALLOC_GROW() here, but\nactually that part is in the context. ;)\n\n> @@ -35,7 +36,9 @@ enum dfs_state {\n> *\n> * \"size\" is the uncompressed object size. Compressed size of the raw\n> * data for an object in a pack is not stored anywhere but is computed\n> - * and made available when reverse .idx is made.\n> + * and made available when reverse .idx is made. Note that when an\n> + * delta is reused, \"size\" is the uncompressed _delta_ size, not the\n> + * canonical one after the delta has been applied.\n\ns/an delta/a delta/\n\n> +Running tests with special setups\n> +---------------------------------\n> +\n> +The whole test suite could be run to test some special features\n> +that cannot be easily covered by a few specific test cases. These\n> +could be enabled by running the test suite with correct GIT_TEST_\n> +environment set.\n> +\n> +GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n> +\n> +GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n> +path where there are more than 1024 packs even if the actual number of\n> +packs in repository is below this limit.\n> +\n> +GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n> +code path where we do not cache objecct size in memory and read it\n> +from existing packs on demand. This normally only happens when the\n> +object size is over 2GB. This variable forces the code path on any\n> +object larger than 2^ bytes.\n\nIt's nice to have these available to test the uncommon cases. But I have\na feeling nobody will ever run them, since it requires extra effort (and\ntakes a full test run).\n\nI see there's a one-off test for GIT_TEST_FULL_IN_PACK_ARRAY, which I\nthink is a good idea, since it makes sure the code is exercised in a\nnormal test suite run. Should we do the same for GIT_TEST_OE_SIZE_BITS?\n\nAlso, s/objecct/object/. :)\n\n> [...]\n\nI haven't done an in-depth read of each patch yet; this was just what\njumped out at me from reading the interdiff.\n\n-Peff\n"},{"id":"343030","messageId":"CACsJy8Br0+PGauEYLUarOFrq4bPbnE5u95XemzFyyn_riBZ_xA@mail.gmail.com","threadId":"47943","inReplyTo":"20180326151317.GA24425@sigill.intra.peff.net","subject":"Re: [PATCH v7 00/13] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-26T17:04:54Z","receivedAt":"2018-03-26T17:05:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 26, 2018 at 5:13 PM, Jeff King wrote:\n> On Sat, Mar 24, 2018 at 07:33:40AM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> +unsigned long oe_get_size_slow(struct packing_data *pack,\n>> + const struct object_entry *e)\n>> +{\n>> + struct packed_git *p;\n>> + struct pack_window *w_curs;\n>> + unsigned char *buf;\n>> + enum object_type type;\n>> + unsigned long used, avail, size;\n>> +\n>> + if (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {\n>> + read_lock();\n>> + if (sha1_object_info(e->idx.oid.hash, &size) < 0)\n>> + die(_(\"unable to get size of %s\"),\n>> + oid_to_hex(&e->idx.oid));\n>> + read_unlock();\n>> + return size;\n>> + }\n>> +\n>> + p = oe_in_pack(pack, e);\n>> + if (!p)\n>> + die(\"BUG: when e->type is a delta, it must belong to a pack\");\n>> +\n>> + read_lock();\n>> + w_curs = NULL;\n>> + buf = use_pack(p, &w_curs, e->in_pack_offset, &avail);\n>> + used = unpack_object_header_buffer(buf, avail, &type, &size);\n>> + if (used == 0)\n>> + die(_(\"unable to parse object header of %s\"),\n>> + oid_to_hex(&e->idx.oid));\n>> +\n>> + unuse_pack(&w_curs);\n>> + read_unlock();\n>> + return size;\n>> +}\n>\n> It took me a while to figure out why this treated deltas and non-deltas\n> differently. At first I thought it was an optimization (since we can\n> find non-delta sizes quickly by looking at the headers). But I think\n> it's just that you want to know the size of the actual _delta_, not the\n> reconstructed object. And there's no way to ask sha1_object_info() for\n> that.\n>\n> Perhaps the _extended version of that function should learn an\n> OBJECT_INFO_NO_DEREF flag or something to tell it return the true delta\n> type and size. Then this whole function could just become a single call.\n>\n> But short of that, it's probably worth a comment explaining what's going\n> on.\n\nI thought the elaboration on \"size\" in the big comment block in front\nof struct object_entry was enough. I was wrong. Will add something\nhere.\n\n>> +Running tests with special setups\n>> +---------------------------------\n>> +\n>> +The whole test suite could be run to test some special features\n>> +that cannot be easily covered by a few specific test cases. These\n>> +could be enabled by running the test suite with correct GIT_TEST_\n>> +environment set.\n>> +\n>> +GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n>> +\n>> +GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n>> +path where there are more than 1024 packs even if the actual number of\n>> +packs in repository is below this limit.\n>> +\n>> +GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n>> +code path where we do not cache objecct size in memory and read it\n>> +from existing packs on demand. This normally only happens when the\n>> +object size is over 2GB. This variable forces the code path on any\n>> +object larger than 2^ bytes.\n>\n> It's nice to have these available to test the uncommon cases. But I have\n> a feeling nobody will ever run them, since it requires extra effort (and\n> takes a full test run).\n\nI know :) I also know that this does not interfere with\nGIT_TEST_SPLIT_INDEX, which is being run in Travis. So the plan (after\nthis series is merged) is to make Travis second run to do something\nlike\n\nmake test GIT_TEST_SPLIT...=1 GIT_TEST_FULL..=1 GIT_TEST_OE..=4\n\nwe don't waste more cpu cycles and we can make sure these code paths\nare always run (at least on one platform)\n\n> I see there's a one-off test for GIT_TEST_FULL_IN_PACK_ARRAY, which I\n> think is a good idea, since it makes sure the code is exercised in a\n> normal test suite run. Should we do the same for GIT_TEST_OE_SIZE_BITS?\n\nI think the problem with OE_SIZE_BITS is it has many different code\npaths (like reused deltas) which is hard to make sure it runs. But yes\nI think I could construct a pack that executes both code paths in\noe_get_size_slow(). Will do in a reroll.\n\n> I haven't done an in-depth read of each patch yet; this was just what\n> jumped out at me from reading the interdiff.\n\nI would really appreciate it if you could find some time to do it. The\nbugs I found in this round proved that I had no idea what's really\ngoing on in pack-objects. Sure I know the big picture but that's far\nfrom enough to do changes like this.\n-- \nDuy\n"},{"id":"343058","messageId":"CAGZ79kamsRrqZ09BtUZPdQEKXNn9X3CphsmUdoqsKv-T=kGSsg@mail.gmail.com","threadId":"47943","inReplyTo":"20180316183200.31014-7-pclouds@gmail.com","subject":"Re: [PATCH v4 06/11] pack-objects: move in_pack out of struct object_entry","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-03-26T20:39:01Z","receivedAt":"2018-03-26T20:39:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi,\n\nsorry for the late review, as I am pointed here indirectly via\nhttps://public-inbox.org/git/xmqqy3iebpsw.fsf@gitster-ct.c.googlers.com/\n\nOn Fri, Mar 16, 2018 at 11:33 AM Nguyễn Thái Ngọc Duy \nwrote:\n\n> +LIMITATIONS\n> +-----------\n> +\n> +This command could only handle 16384 existing pack files at a time.\n\ns/could/can/ ?\n\n> @@ -3191,6 +3200,9 @@ int cmd_pack_objects(int argc, const char **argv,\nconst char *prefix)\n> }\n> }\n\n> + /* make sure IN_PACK(0) return NULL */\n\nI was confused for a while staring at this comment, /s/0/NULL/\nwould have helped me.\n\n> +static inline unsigned int oe_add_pack(struct packing_data *pack,\n> + struct packed_git *p)\n> +{\n> + if (pack->in_pack_count >= (1 << OE_IN_PACK_BITS))\n> + die(_(\"too many packs to handle in one go. \"\n> + \"Please add .keep files to exclude\\n\"\n> + \"some pack files and keep the number \"\n> + \"of non-kept files below %d.\"),\n> + 1 << OE_IN_PACK_BITS);\n\nThe packs are indexed 0..N-1, so we can actually handle N\npacks I presume. But if we actually have N, then we'd run the\n\n /* make sure IN_PACK(0) return NULL */\n oe_add_pack(.., NULL);\n\nas N+1, hence the user can only do N-1 ?\n\nOh wait! the code below makes me think we index from 1..N,\ntreating index 0 special as uninitialized? So we actually can only\nstore N-1 ?\n\n\n> + if (p) {\n> + if (p->index > 0)\n\ns/>/!=/ ?\n\nThe new index variable is only used in these three\ninlined header functions, and in_pack_count is strictly\npositive, so index as well as in_pack_count could be made\nunsigned?\n\nGiven that oe_add_pack returns an unsigned, I would actually\nprefer to have in_pack_count an unsigned as well.\n\n> + die(\"BUG: this packed is already indexed\");\n> + p->index = pack->in_pack_count;\n> + }\n> + pack->in_pack[pack->in_pack_count] = p;\n> + return pack->in_pack_count++;\n> +}\n> +\n> +static inline struct packed_git *oe_in_pack(const struct packing_data\n*pack,\n> + const struct object_entry *e)\n> +{\n> + return pack->in_pack[e->in_pack_idx];\n> +\n> +}\n\nextra new line after return?\n\n> +static inline void oe_set_in_pack(struct object_entry *e,\n> + struct packed_git *p)\n> +{\n> + if (p->index <= 0)\n> + die(\"BUG: found_pack should be NULL \"\n> + \"instead of having non-positive index\");\n\nDo we also want to guard against\n p->index > (1 << OE_IN_PACK_BITS)\nhere? Also there is a BUG() macro, that would be better\nas it reports line file/number, but we cannot use it here as\nit is a header inline.\n"},{"id":"343150","messageId":"20180327165311.GA15102@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8Br0+PGauEYLUarOFrq4bPbnE5u95XemzFyyn_riBZ_xA@mail.gmail.com","subject":"Re: [PATCH v7 00/13] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-27T16:53:11Z","receivedAt":"2018-03-27T16:53:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 26, 2018 at 07:04:54PM +0200, Duy Nguyen wrote:\n\n> >> +unsigned long oe_get_size_slow(struct packing_data *pack,\n> >> + const struct object_entry *e)\n> [...]\n> > But short of that, it's probably worth a comment explaining what's going\n> > on.\n> \n> I thought the elaboration on \"size\" in the big comment block in front\n> of struct object_entry was enough. I was wrong. Will add something\n> here.\n\nIt may be my fault for reading the interdiff, which didn't include that\ncomment. I was literally just thinking something like:\n\n /*\n * Return the size of the object without doing any delta\n * reconstruction (so non-deltas are true object sizes, but\n * deltas return the size of the delta data).\n */\n\n> > I see there's a one-off test for GIT_TEST_FULL_IN_PACK_ARRAY, which I\n> > think is a good idea, since it makes sure the code is exercised in a\n> > normal test suite run. Should we do the same for GIT_TEST_OE_SIZE_BITS?\n> \n> I think the problem with OE_SIZE_BITS is it has many different code\n> paths (like reused deltas) which is hard to make sure it runs. But yes\n> I think I could construct a pack that executes both code paths in\n> oe_get_size_slow(). Will do in a reroll.\n\nOK. If it's too painful to construct a good example, don't worry about\nit. It sounds like we're unlikely to get full coverage anyway.\n\n> > I haven't done an in-depth read of each patch yet; this was just what\n> > jumped out at me from reading the interdiff.\n> \n> I would really appreciate it if you could find some time to do it. The\n> bugs I found in this round proved that I had no idea what's really\n> going on in pack-objects. Sure I know the big picture but that's far\n> from enough to do changes like this.\n\nI didn't get to it today, but I'll try to give it a careful read. There\nare quite a few corners of pack-objects I don't know well, but I think\nat this point I may be the most expert of remaining people. Scary. :)\n\n-Peff\n"},{"id":"343474","messageId":"20180330201832.GA14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-3-pclouds@gmail.com","subject":"Re: [PATCH v7 02/13] pack-objects: turn type and in_pack_type to bitfields","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:18:32Z","receivedAt":"2018-03-30T20:18:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:42AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> +static inline void oe_set_type(struct object_entry *e,\n> +\t\t\t enum object_type type)\n> +{\n> +\tif (type >= OBJ_ANY)\n> +\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n\nA minor nit, but this (and other new assertions) should probably be\nBUG().\n\n> +\te->type_valid = type >= OBJ_NONE;\n> +\te->type_ = (unsigned)type;\n\nHmm, so if !e->type_valid, then we may write utter garbage into\ne->type_. That's OK, since everybody will access it via oe_type(), but I\nwonder if we could trigger weird compiler behavior. I guess the unsigned\ncast makes the truncation well-defined.\n\n-Peff\n"},{"id":"343475","messageId":"20180330202347.GB14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-4-pclouds@gmail.com","subject":"Re: [PATCH v7 03/13] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:23:47Z","receivedAt":"2018-03-30T20:23:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:43AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\nThis probably needs some explanation for people digging in history (even\nif it's \"this is to shrink the size as part of a larger struct-shrinking\neffort\" so they know to dig around in the nearby history).\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 647c01ea34..83f8154865 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\tOPT_END(),\n> \t};\n> \n> +\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n> +\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n\nI thought this was off-by-one at first, but NUM_STATES is one more than\nthe highest state, so it's right.\n\nI suspect all of the dfs and depth stuff could be pulled into a separate\narray that is used only during that depth search. But as you have it\nsquished down here, I think we may be getting it \"for free\" in between\nother non-word-aligned values in the struct.\n\n-Peff\n"},{"id":"343478","messageId":"20180330202645.GC14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-5-pclouds@gmail.com","subject":"Re: [PATCH v7 04/13] pack-objects: use bitfield for object_entry::depth","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:26:45Z","receivedAt":"2018-03-30T20:26:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:44AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> Because of struct packing from now on we can only handle max depth\n> 4095 (or even lower when new booleans are added in this struct). This\n> should be ok since long delta chain will cause significant slow down\n> anyway.\n\nOK. This is the first user-facing change, but I think it really\nshouldn't hurt anybody. My experiments a while ago showed that chains\nlonger than 50 aren't really worth it, but so this could probably shrink\nto something like 8 bits if we really needed it to.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 83f8154865..205e1f646c 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3068,6 +3068,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (pack_to_stdout != !base_name || argc)\n> \t\tusage_with_options(pack_usage, pack_objects_options);\n> \n> +\tif (depth >= (1 << OE_DEPTH_BITS))\n> +\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n> +\t\t depth, (1 << OE_DEPTH_BITS) - 1);\n\nSince this is introducing a new limit, I wonder if we should issue a\nwarning and just clamp it to the maximum value. That would be kinder to\npeople who may have existing (admittedly dumb) setups.\n\n-Peff\n"},{"id":"343479","messageId":"20180330203045.GD14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-6-pclouds@gmail.com","subject":"Re: [PATCH v7 05/13] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:30:46Z","receivedAt":"2018-03-30T20:30:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:45AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> This field is only need for pack-bitmap, which is an optional\n> feature. Move it to a separate array that is only allocated when\n> pack-bitmap is used (it's not freed in the same way that objects[] is\n> not).\n\nI had trouble parsing the parenthetical in the last sentence. It does\nmake sense if you read it hard enough, but maybe:\n\n (like objects[], it is not freed, since we need it until the end of\n the process)\n\nwould be more clear?\n\nThe patch itself seems OK.\n\n-Peff\n"},{"id":"343481","messageId":"20180330204812.GE14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-7-pclouds@gmail.com","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:48:12Z","receivedAt":"2018-03-30T20:48:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:46AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index e1244918a5..b41610569e 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -29,6 +29,8 @@\n> #include \"list.h\"\n> #include \"packfile.h\"\n> \n> +#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n\nHow come this one gets a macro, but the earlier conversions don't?\n\nI guess the problem is that oe_in_pack() is defined in the generic\npack-objects.h, but &to_pack is only in builtin/pack-objects.c?\n\nI wonder if it would be that bad to just say oe_in_pack(&to_pack, obj)\neverywhere. It's longer, but it makes the code slightly less magical to\nread.\n\n> @@ -1074,7 +1076,7 @@ static void create_object_entry(const struct object_id *oid,\n> \telse\n> \t\tnr_result++;\n> \tif (found_pack) {\n> -\t\tentry->in_pack = found_pack;\n> +\t\toe_set_in_pack(&to_pack, entry, found_pack);\n> \t\tentry->in_pack_offset = found_offset;\n> \t}\n\nit's funny to see in_pack as an external thing, but in_pack_offset still\nin the struct. I guess there's nothing to be gained there, since the\noffset really does need to be individual (and large).\n\n> diff --git a/cache.h b/cache.h\n> index 862bdff83a..b90feb3802 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1635,6 +1635,7 @@ extern struct packed_git {\n> \tint index_version;\n> \ttime_t mtime;\n> \tint pack_fd;\n> +\tint index;\t\t/* for builtin/pack-objects.c */\n> \tunsigned pack_local:1,\n> \t\t pack_keep:1,\n> \t\t freshened:1,\n\nIt's pretty gross to infect this global struct. But I'm not sure there's\nan easier way to do it with constant-time lookups. You'd have to build\nthe packed_git index preemptively in pack-objects, and then always just\npass around the index numbers. And even that is kind of dicey, since\nthe packed_git list can grow while we're running.\n\nThe alternative is a hash table mapping packed_git pointers into numeric\nindices. Yuck.\n\n> +static void prepare_in_pack_by_idx(struct packing_data *pdata)\n> +{\n> +\tstruct packed_git **mapping, *p;\n> +\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n> +\n> +\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n> +\t\t/*\n> +\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n> +\t\t * used instead\n> +\t\t */\n> +\t\treturn;\n> +\t}\n> +\n> +\tALLOC_ARRAY(mapping, nr);\n> +\tmapping[cnt++] = NULL; /* zero index must be mapped to NULL */\n\nWhy? I guess because index==0 is a sentinel for \"we're using the small\nindex numbers?\"\n\n> +\tprepare_packed_git();\n> +\tfor (p = packed_git; p; p = p->next, cnt++) {\n> +\t\tif (cnt == nr) {\n> +\t\t\tfree(mapping);\n> +\t\t\treturn;\n> +\t\t}\n> +\t\tp->index = cnt;\n> +\t\tmapping[cnt] = p;\n> +\t}\n> +\tpdata->in_pack_by_idx = mapping;\n> +}\n\nWhat happens if we later have to reprepare_packed_git() and end up with\nmore packs? We only call this for the first pack.\n\nIt may well be handled, but I'm having trouble following the code to see\nif it is. And I doubt that case is covered by our test suite (since it\ninherently involves a race).\n\n> /*\n> + * The size of struct nearly determines pack-objects's memory\n> + * consumption. This struct is packed tight for that reason. When you\n> + * add or reorder something in this struct, think a bit about this.\n> + *\n\nIt's funny to see this warning come in the middle. Should it be part of\nthe final struct reordering at the end?\n\n-Peff\n"},{"id":"343482","messageId":"20180330205346.GF14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-8-pclouds@gmail.com","subject":"Re: [PATCH v7 07/13] pack-objects: refer to delta objects by index instead of pointer","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:53:46Z","receivedAt":"2018-03-30T20:54:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:47AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> These delta pointers always point to elements in the objects[] array\n> in packing_data struct. We can only hold maximum 4G of those objects\n> because the array size in nr_objects is uint32_t. We could use\n> uint32_t indexes to address these elements instead of pointers. On\n> 64-bit architecture (8 bytes per pointer) this would save 4 bytes per\n> pointer.\n> \n> Convert these delta pointers to indexes. Since we need to handle NULL\n> pointers as well, the index is shifted by one [1].\n> \n> [1] This means we can only index 2^32-2 objects even though nr_objects\n> could contain 2^32-1 objects. It should not be a problem in\n> practice because when we grow objects[], nr_alloc would probably\n> blow up long before nr_objects hits the wall.\n\nHmm, that may be something we eventually fix. I suspect all of this code\ndoes some pretty horrible things as you approach 2^32 objects, though.\nI've never tried to make such a pack, but it may be within the realm of\npossibility. The .idx file would be 80+GB, but the packfile might not be\nmuch bigger if specially crafted.\n\nI guess that's outside the realm of reasonable, though, so we can assume\nthat nobody would _really_ want to do that anytime soon. And anything\nmalicious would probably die long before this code triggers.\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n> pack-objects.h | 67 ++++++++++++++++++++++--\n> 2 files changed, 124 insertions(+), 59 deletions(-)\n\nThe patch itself looks OK. This is one of the nicer ones, because it\nreally doesn't involve any extra storage management, just some accessor\nfunctions.\n\n-Peff\n"},{"id":"343484","messageId":"20180330205905.GG14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-9-pclouds@gmail.com","subject":"Re: [PATCH v7 08/13] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T20:59:06Z","receivedAt":"2018-03-30T20:59:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:48AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> We only cache deltas when it's smaller than a certain limit. This limit\n> defaults to 1000 but save its compressed length in a 64-bit field.\n> Shrink that field down to 16 bits, so you can only cache 65kb deltas.\n> Larger deltas must be recomputed at when the pack is written down.\n\nUnlike the depth, I don't think there's any _inherent_ reason you\ncouldn't throw, say, 1MB deltas into the cache (if you sized it large\nenough). But I doubt such deltas are really all that common. Here are\nthe top 10 in linux.git:\n\n $ git cat-file --batch-all-objects --batch-check='%(deltabase) %(objectsize:disk)' |\n grep -v ^00000 | sort -k 2nr | head\n a02b6794337286bc12c907c33d5d75537c240bd0 769103\n b28d4b64c05da02c5e8c684dcb9422876225ebdc 327116\n 1e98ce86ed19aff9ba721d13a749ff08088c9922 325257\n a02b6794337286bc12c907c33d5d75537c240bd0 240647\n c550d99286c01867dfb26e432417f3106acf8611 177896\n 5977795854f852c2b95dd023fd03cace023ee41c 119737\n 4ccf9681c45d01d17376f7e0d266532a4460f5f8 112671\n b39fb6821faa9e7bc36de738152a2817b4bf3654 112657\n 2645d6239b74bebd661436762e819b831095b084 103980\n b8ce7fe5d8def58dc63b7ae099eff7bd07e4e845 101014\n\nIt's possible some weird workload would want to tweak this. Say you were\nstoring a ton of delta-capable files that were big and always differed\nby a megabyte. And it was somehow really important to you to tradeoff\nmemory for CPU during the write phase of a pack.\n\nThat seems pretty unlikely to bite anybody (and that was the best I\ncould come up with as a devil's advocate against it).\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/config.txt | 3 ++-\n> builtin/pack-objects.c | 22 ++++++++++++++++------\n> pack-objects.h | 3 ++-\n> 3 files changed, 20 insertions(+), 8 deletions(-)\n\nPatch looks OK.\n\n-Peff\n"},{"id":"343485","messageId":"20180330210416.GH14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-11-pclouds@gmail.com","subject":"Re: [PATCH v7 10/13] pack-objects: clarify the use of object_entry::size","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T21:04:17Z","receivedAt":"2018-03-30T21:04:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:50AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> While this field most of the time contains the canonical object size,\n> there is one case it does not: when we have found that the base object\n> of the delta in question is also to be packed, we will very happily\n> reuse the delta by copying it over instead of regenerating the new\n> delta.\n> \n> \"size\" in this case will record the delta size, not canonical object\n> size. Later on in write_reuse_object(), we reconstruct the delta\n> header and \"size\" is used for this purpose. When this happens, the\n> \"type\" field contains a delta type instead of a canonical type.\n> Highlight this in the code since it could be tricky to see.\n\nThanks for digging down here. I have definitely been confused by this in\nthe past.\n\nThe subject says \"clarify\" so I was a little surprised to see code\nchanges. It looks like we're just avoiding reassigning on top of the\nvalue repeatedly, which is part of that clarification. It looks like a\nnoop to me.\n\n-Peff\n"},{"id":"343486","messageId":"20180330211844.GI14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-12-pclouds@gmail.com","subject":"Re: [PATCH v7 11/13] pack-objects: shrink size field in struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T21:18:44Z","receivedAt":"2018-03-30T21:18:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:51AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> It's very very rare that an uncompressed object is larger than 4GB\n> (partly because Git does not handle those large files very well to\n> begin with). Let's optimize it for the common case where object size\n> is smaller than this limit.\n> \n> Shrink size field down to 32 bits [1] and one overflow bit. If the\n> size is too large, we read it back from disk. As noted in the previous\n> patch, we need to return the delta size instead of canonical size when\n> the to-be-reused object entry type is a delta instead of a canonical\n> one.\n> \n> Add two compare helpers that can take advantage of the overflow\n> bit (e.g. if the file is 4GB+, chances are it's already larger than\n> core.bigFileThreshold and there's no point in comparing the actual\n> value).\n> \n> Another note about oe_get_size_slow(). This function MUST be thread\n> safe because SIZE() macro is used inside try_delta() which may run in\n> parallel. Outside parallel code, no-contention locking should be dirt\n> cheap (or insignificant compared to i/o access anyway). To exercise\n> this code, it's best to run the test suite with something like\n> \n> make test GIT_TEST_OE_SIZE_BITS=2\n> \n> which forces this code on all objects larger than 3 bytes.\n\nOK, makes sense. Since we need it to be thread-safe, we have to use\nread_lock(). Which means that oe_get_size_slow() is defined in\nbuiltin/pack-objects.c. But the object_entry is defined in the\nmore-generic pack-objects.h.\n\nSo anybody besides builtin/pack-objects.c will have to implement their\nown fallback when e->size_valid isn't true. Which is a little odd, but I\nguess nobody else needs that field. It might bite us in the future, but\nI'm willing to cross my fingers for now (the pack-objects.h header is\nreally just there to support the bitmap writing code, but even that\ncould in theory all get shoved into a single translation unit if we had\nto).\n\n> [1] it's actually already 32 bits on Windows\n\nAnd linux-i386. :)\n\n> +unsigned long oe_get_size_slow(struct packing_data *pack,\n> +\t\t\t const struct object_entry *e)\n> +{\n\nI think I already replied about this earlier, so I'll skim over it this\ntime.\n\n> diff --git a/pack-objects.c b/pack-objects.c\n> index 13f2b2bff2..59c6e40a02 100644\n> --- a/pack-objects.c\n> +++ b/pack-objects.c\n> @@ -120,8 +120,15 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n> {\n> \tstruct object_entry *new_entry;\n> \n> -\tif (!pdata->nr_objects)\n> +\tif (!pdata->nr_objects) {\n> \t\tprepare_in_pack_by_idx(pdata);\n> +\t\tif (getenv(\"GIT_TEST_OE_SIZE_BITS\")) {\n> +\t\t\tint bits = atoi(getenv(\"GIT_TEST_OE_SIZE_BITS\"));;\n> +\t\t\tpdata->oe_size_limit = 1 << bits;\n> +\t\t}\n> +\t\tif (!pdata->oe_size_limit)\n> +\t\t\tpdata->oe_size_limit = 1 << OE_SIZE_BITS;\n> +\t}\n\nThis needs to be \"1U << OE_SIZE_BITS\". Shifting a signed integer 31 bits\nis undefined.\n\nNo, I'm not that clever or careful myself. I ran the whole test suite\nwith SANITIZE=address,undefined and it turned this up, as well as a\nsimilar case for OE_DELTA_SIZE_BITS.\n\n> +\tuint32_t size_:OE_SIZE_BITS;\n> +\tuint32_t size_valid:1;\n\nA uint32_t bitfield? Would it make more sense to just call these\n\"unsigned\", since we're specifying the precision already?\n\n> +unsigned long oe_get_size_slow(struct packing_data *pack,\n> +\t\t\t const struct object_entry *e);\n> +static inline unsigned long oe_size(struct packing_data *pack,\n> +\t\t\t\t const struct object_entry *e)\n> +{\n> +\tif (e->size_valid)\n> +\t\treturn e->size_;\n> +\n> +\treturn oe_get_size_slow(pack, e);\n> +}\n\nIf oe_get_size_slow() fails to find an object's size, it dies. I'm\ntrying to think of whether that might hit funny corner cases with\nracing. I don't _think_ so, because if the object truly goes away, we'd\nbe screwed during the writing phase anyway.\n\n> +static inline int oe_size_less_than(struct packing_data *pack,\n> +\t\t\t\t const struct object_entry *lhs,\n> +\t\t\t\t unsigned long rhs)\n> +{\n> +\tif (lhs->size_valid)\n> +\t\treturn lhs->size_ < rhs;\n> +\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n> +\t\treturn 0;\n> +\treturn oe_get_size_slow(pack, lhs) < rhs;\n> +}\n\nClever.\n\n> +static inline void oe_set_size(struct packing_data *pack,\n> +\t\t\t struct object_entry *e,\n> +\t\t\t unsigned long size)\n> +{\n> +\tif (size < pack->oe_size_limit) {\n> +\t\te->size_ = size;\n> +\t\te->size_valid = 1;\n> +\t} else {\n> +\t\te->size_valid = 0;\n> +\t\tif (oe_get_size_slow(pack, e) != size)\n> +\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n> +\t}\n> +}\n\nThat's an expensive assertion. But I guess this isn't supposed to happen\nvery frequently, so it's probably OK.\n\n-Peff\n"},{"id":"343487","messageId":"20180330212419.GJ14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-13-pclouds@gmail.com","subject":"Re: [PATCH v7 12/13] pack-objects: shrink delta_size field in struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T21:24:20Z","receivedAt":"2018-03-30T21:24:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:52AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> Allowing a delta size of 64 bits is crazy. Shrink this field down to\n> 31 bits with one overflow bit.\n> \n> If we find an existing delta larger than 2GB, we do not cache\n> delta_size at all and will get the value from oe_size(), potentially\n> from disk if it's larger than 4GB.\n\nSince we have a fallback, we can put this slider wherever we want.\nProbably something like 20 bits would be plenty, if we ever needed to\nsqueeze in a few more small-bit items.\n\n> @@ -2004,10 +2006,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n> \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n> \tif (!delta_buf)\n> \t\treturn 0;\n> +\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n> +\t\treturn 0;\n\nThis is the other spot that needs to be \"1U\".\n\nHow come this doesn't get a pdata->oe_delta_size_limit like we have\npdata->oe_size_limit? Would we want a matching\n$GIT_TEST_OE_DELTA_SIZE_BITS to test it, too?\n\n-Peff\n"},{"id":"343488","messageId":"20180330212656.GK14782@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180324063353.24722-14-pclouds@gmail.com","subject":"Re: [PATCH v7 13/13] pack-objects: reorder members to shrink struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-30T21:26:56Z","receivedAt":"2018-03-30T21:27:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 24, 2018 at 07:33:53AM +0100, Nguyễn Thái Ngọc Duy wrote:\n\n> Previous patches leave lots of holes and padding in this struct. This\n> patch reorders the members and shrinks the struct down to 80 bytes\n> (from 136 bytes, before any field shrinking is done) with 16 bits to\n> spare (and a couple more in in_pack_header_size when we really run out\n> of bits).\n\nOut of curiosity, did you count this yourself, or did you double-check\nwith a few compilers to make sure they all produce the same result?\n\nSo having read the whole thing now, I think most of my original concerns\nhave been addressed. I do think readability takes a hit, but it's not\n_too_ bad. There are a few things that have become more brittle, but I\ncan't think of anything on the horizon that would bite us.\n\n-Peff\n"},{"id":"343496","messageId":"CACsJy8Cr=uN8d9ZWO=xbyOYbMmS6V3FCdw_tww15d6NHpLMTag@mail.gmail.com","threadId":"47943","inReplyTo":"20180330212656.GK14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 13/13] pack-objects: reorder members to shrink struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T04:10:30Z","receivedAt":"2018-03-31T04:11:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 11:26 PM, Jeff King wrote:\n> On Sat, Mar 24, 2018 at 07:33:53AM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> Previous patches leave lots of holes and padding in this struct. This\n>> patch reorders the members and shrinks the struct down to 80 bytes\n>> (from 136 bytes, before any field shrinking is done) with 16 bits to\n>> spare (and a couple more in in_pack_header_size when we really run out\n>> of bits).\n>\n> Out of curiosity, did you count this yourself, or did you double-check\n> with a few compilers to make sure they all produce the same result?\n\nI used pahole though only with .o files created by gcc 64-bit. I'll\ntry the 32-bit version and clang as well.\n-- \nDuy\n"},{"id":"343497","messageId":"CACsJy8Bt6eY5vu4SRvnrY3Wq3zC3j9yALXJy2Y1eL=LVrMickw@mail.gmail.com","threadId":"47943","inReplyTo":"20180330212419.GJ14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 12/13] pack-objects: shrink delta_size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T04:21:37Z","receivedAt":"2018-03-31T04:22:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 11:24 PM, Jeff King wrote:\n> On Sat, Mar 24, 2018 at 07:33:52AM +0100, Nguyễn Thái Ngọc Duy wrote:\n>> @@ -2004,10 +2006,12 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n>> delta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n>> if (!delta_buf)\n>> return 0;\n>> + if (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n>> + return 0;\n>\n> This is the other spot that needs to be \"1U\".\n>\n> How come this doesn't get a pdata->oe_delta_size_limit like we have\n> pdata->oe_size_limit? Would we want a matching\n> $GIT_TEST_OE_DELTA_SIZE_BITS to test it, too?\n\nProbably. This change does not look as risky as the others (no\ncomplicated fallback). But without $GIT_TEST_OE_DELTA_SIZE_BITS it's\nhard to know how the new code reacts when we get over the limit. I\nwill add it.\n-- \nDuy\n"},{"id":"343498","messageId":"CACsJy8AQ7yRK65rYqMkR+QJn2KUJ=Y+9E_=k3aBU78xF=LBQjw@mail.gmail.com","threadId":"47943","inReplyTo":"20180330210416.GH14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 10/13] pack-objects: clarify the use of object_entry::size","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T04:35:40Z","receivedAt":"2018-03-31T04:36:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 11:04 PM, Jeff King wrote:\n> The subject says \"clarify\" so I was a little surprised to see code\n> changes. It looks like we're just avoiding reassigning on top of the\n> value repeatedly, which is part of that clarification. It looks like a\n> noop to me.\n\nOh well... I was counting on the new name (in_pack_size, which follows\nin_pack_type naming convention) to emphasize it (and the new \"delta\nsize\" comment to point out where in_pack_size contains a delta size.\n-- \nDuy\n"},{"id":"343499","messageId":"CACsJy8DFpqa5DYmNpBpeWOW1a3UH64AHa2vG3nkJVLoMR73wqQ@mail.gmail.com","threadId":"47943","inReplyTo":"20180330205905.GG14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 08/13] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T04:40:23Z","receivedAt":"2018-03-31T04:40:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 10:59 PM, Jeff King wrote:\n> On Sat, Mar 24, 2018 at 07:33:48AM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> We only cache deltas when it's smaller than a certain limit. This limit\n>> defaults to 1000 but save its compressed length in a 64-bit field.\n>> Shrink that field down to 16 bits, so you can only cache 65kb deltas.\n>> Larger deltas must be recomputed at when the pack is written down.\n>\n> Unlike the depth, I don't think there's any _inherent_ reason you\n> couldn't throw, say, 1MB deltas into the cache (if you sized it large\n> enough). But I doubt such deltas are really all that common. Here are\n> the top 10 in linux.git:\n>\n> $ git cat-file --batch-all-objects --batch-check='%(deltabase) %(objectsize:disk)' |\n> grep -v ^00000 | sort -k 2nr | head\n> a02b6794337286bc12c907c33d5d75537c240bd0 769103\n> b28d4b64c05da02c5e8c684dcb9422876225ebdc 327116\n> 1e98ce86ed19aff9ba721d13a749ff08088c9922 325257\n> a02b6794337286bc12c907c33d5d75537c240bd0 240647\n> c550d99286c01867dfb26e432417f3106acf8611 177896\n> 5977795854f852c2b95dd023fd03cace023ee41c 119737\n> 4ccf9681c45d01d17376f7e0d266532a4460f5f8 112671\n> b39fb6821faa9e7bc36de738152a2817b4bf3654 112657\n> 2645d6239b74bebd661436762e819b831095b084 103980\n> b8ce7fe5d8def58dc63b7ae099eff7bd07e4e845 101014\n>\n> It's possible some weird workload would want to tweak this. Say you were\n> storing a ton of delta-capable files that were big and always differed\n> by a megabyte. And it was somehow really important to you to tradeoff\n> memory for CPU during the write phase of a pack.\n\nWe're not short on spare bits so I will try to raise this limit to 1MB\n(not because you mentioned 1MB, but because the largest size in your\noutput is close to 1MB).\n-- \nDuy\n"},{"id":"343500","messageId":"CACsJy8Dma-xrDcK1vQ-s4F+Dwt4oWOQ89yr33OPJdO1g0ryQWw@mail.gmail.com","threadId":"47943","inReplyTo":"20180330204812.GE14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T04:51:10Z","receivedAt":"2018-03-31T04:51:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 10:48 PM, Jeff King wrote:\n> On Sat, Mar 24, 2018 at 07:33:46AM +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index e1244918a5..b41610569e 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -29,6 +29,8 @@\n>> #include \"list.h\"\n>> #include \"packfile.h\"\n>>\n>> +#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n>\n> How come this one gets a macro, but the earlier conversions don't?\n>\n> I guess the problem is that oe_in_pack() is defined in the generic\n> pack-objects.h, but &to_pack is only in builtin/pack-objects.c?\n>\n> I wonder if it would be that bad to just say oe_in_pack(&to_pack, obj)\n> everywhere. It's longer, but it makes the code slightly less magical to\n> read.\n\nLonger was exactly why I added these macros (with the hope that the\nmacro upper case names already ring a \"it's magical\" bell). Should I\ndrop all these macros? Some code becomes a lot more verbose though.\n\n>> +static void prepare_in_pack_by_idx(struct packing_data *pdata)\n>> +{\n>> + struct packed_git **mapping, *p;\n>> + int cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n>> +\n>> + if (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n>> + /*\n>> + * leave in_pack_by_idx NULL to force in_pack[] to be\n>> + * used instead\n>> + */\n>> + return;\n>> + }\n>> +\n>> + ALLOC_ARRAY(mapping, nr);\n>> + mapping[cnt++] = NULL; /* zero index must be mapped to NULL */\n>\n> Why? I guess because index==0 is a sentinel for \"we're using the small\n> index numbers?\"\n\nNo because by default all values in object_entry is zero (or NULL). If\nI remember correctly, some code will skip setting in_pack pointer to\nleave it NULL. When we convert it to an index, it should also point to\nNULL.\n\n>> + prepare_packed_git();\n>> + for (p = packed_git; p; p = p->next, cnt++) {\n>> + if (cnt == nr) {\n>> + free(mapping);\n>> + return;\n>> + }\n>> + p->index = cnt;\n>> + mapping[cnt] = p;\n>> + }\n>> + pdata->in_pack_by_idx = mapping;\n>> +}\n>\n> What happens if we later have to reprepare_packed_git() and end up with\n> more packs? We only call this for the first pack.\n>\n> It may well be handled, but I'm having trouble following the code to see\n> if it is. And I doubt that case is covered by our test suite (since it\n> inherently involves a race).\n\nI don't think I covered this case. But since \"index\" field in\npacked_git should be zero for the new packs, we could check and either\nadd it to in_pack_by_idx[].\n\n>> /*\n>> + * The size of struct nearly determines pack-objects's memory\n>> + * consumption. This struct is packed tight for that reason. When you\n>> + * add or reorder something in this struct, think a bit about this.\n>> + *\n>\n> It's funny to see this warning come in the middle. Should it be part of\n> the final struct reordering at the end?\n\nIt was at the end in some version, the I shuffled the patches and\nforgot about this one :)\n-- \nDuy\n"},{"id":"343502","messageId":"CACsJy8AzW4nkRpHHdS=BJSu1m5rUSL_F+WU719MYB=4TPTAuoA@mail.gmail.com","threadId":"47943","inReplyTo":"20180330212419.GJ14782@sigill.intra.peff.net","subject":"Re: [PATCH v7 12/13] pack-objects: shrink delta_size field in struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T09:10:14Z","receivedAt":"2018-03-31T09:10:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 30, 2018 at 11:24 PM, Jeff King wrote:\n> How come this doesn't get a pdata->oe_delta_size_limit like we have\n> pdata->oe_size_limit? Would we want a matching\n> $GIT_TEST_OE_DELTA_SIZE_BITS to test it, too?\n\nNope. This changes how the delta chain is formed (e.g. produces\nshorter chains) and apparently some tests rely on that, like t5303.\n-- \nDuy\n"},{"id":"343504","messageId":"20180331100311.32373-2-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 01/15] t/README: mention about running the test suite in special modes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:02:57Z","receivedAt":"2018-03-31T10:03:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"From: Duy Nguyen \n\nThere are features that would benefit from running the whole test\nsuite instead of just a few test cases written specifically for\nthem. Split-index mode is one of them. Document it.\n\nThis step is required because a few patches later, we will be\nintroduce more test modes like this to test some corner cases of\npack-objects as much as possible.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/README | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/README b/t/README\nindex 1a1361a806..09eb2b9768 100644\n--- a/t/README\n+++ b/t/README\n@@ -292,6 +292,16 @@ and know what setup is needed for it. Or when you want to run\n everything up to a certain test.\n \n \n+Running tests with special setups\n+---------------------------------\n+\n+The whole test suite could be run to test some special features\n+that cannot be easily covered by a few specific test cases. These\n+could be enabled by running the test suite with correct GIT_TEST_\n+environment set.\n+\n+GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n+\n Naming Tests\n ------------\n \n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343505","messageId":"20180331100311.32373-3-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 02/15] pack-objects: a bit of document about struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:02:58Z","receivedAt":"2018-03-31T10:03:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The role of this comment block becomes more important after we shuffle\nfields around to shrink this struct. It will be much harder to see what\nfield is related to what.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..c0a1f61aac 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,51 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+/*\n+ * basic object info\n+ * -----------------\n+ * idx.oid is filled up before delta searching starts. idx.crc32 is\n+ * only valid after the object is written out and will be used for\n+ * generating the index. idx.offset will be both gradually set and\n+ * used in writing phase (base objects get offset first, then deltas\n+ * refer to them)\n+ *\n+ * \"size\" is the uncompressed object size. Compressed size of the raw\n+ * data for an object in a pack is not stored anywhere but is computed\n+ * and made available when reverse .idx is made.\n+ *\n+ * \"hash\" contains a path name hash which is used for sorting the\n+ * delta list and also during delta searching. Once prepare_pack()\n+ * returns it's no longer needed.\n+ *\n+ * source pack info\n+ * ----------------\n+ * The (in_pack, in_pack_offset) tuple contains the location of the\n+ * object in the source pack. in_pack_header_size allows quickly\n+ * skipping the header and going straight to the zlib stream.\n+ *\n+ * \"type\" and \"in_pack_type\" both describe object type. in_pack_type\n+ * may contain a delta type, while type is always the canonical type.\n+ *\n+ * deltas\n+ * ------\n+ * Delta links (delta, delta_child and delta_sibling) are created to\n+ * reflect that delta graph from the source pack then updated or added\n+ * during delta searching phase when we find better deltas.\n+ *\n+ * delta_child and delta_sibling are last needed in\n+ * compute_write_order(). \"delta\" and \"delta_size\" must remain valid\n+ * at object writing phase in case the delta is not cached.\n+ *\n+ * If a delta is cached in memory and is compressed, delta_data points\n+ * to the data and z_delta_size contains the compressed size. If it's\n+ * uncompressed [1], z_delta_size must be zero. delta_size is always\n+ * the uncompressed size and must be valid even if the delta is not\n+ * cached.\n+ *\n+ * [1] during try_delta phase we don't bother with compressing because\n+ * the delta could be quickly replaced with a better one.\n+ */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343506","messageId":"20180331100311.32373-8-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 07/15] pack-objects: move in_pack out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:03Z","receivedAt":"2018-03-31T10:03:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Instead of using 8 bytes (on 64 bit arch) to store a pointer to a\npack. Use an index instead since the number of packs should be\nrelatively small.\n\nThis limits the number of packs we can handle to 1k. Since we can't be\nsure people can never run into the situation where they have more than\n1k pack files. Provide a fall back route for it.\n\nIf we find out they have too many packs, the new in_pack_by_idx[]\narray (which has at most 1k elements) will not be used. Instead we\nallocate in_pack[] array that holds nr_objects elements. This is\nsimilar to how the optional in_pack_pos field is handled.\n\nThe new simple test is just to make sure the too-many-packs code path\nis at least executed. The true test is running\n\n make test GIT_TEST_FULL_IN_PACK_ARRAY=1\n\nto take advantage of other special case tests.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 28 +++++++++++-------\n cache.h | 1 +\n pack-objects.c | 66 ++++++++++++++++++++++++++++++++++++++++++\n pack-objects.h | 36 ++++++++++++++++++++++-\n t/README | 4 +++\n t/t5300-pack-object.sh | 5 ++++\n 6 files changed, 128 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7a672366bf..4e5812a053 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,6 +29,8 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n+#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -367,7 +369,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = entry->in_pack;\n+\tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -478,7 +480,7 @@ static off_t write_object(struct hashfile *f,\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!entry->in_pack)\n+\telse if (!IN_PACK(entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -1074,7 +1076,7 @@ static void create_object_entry(const struct object_id *oid,\n \telse\n \t\tnr_result++;\n \tif (found_pack) {\n-\t\tentry->in_pack = found_pack;\n+\t\toe_set_in_pack(&to_pack, entry, found_pack);\n \t\tentry->in_pack_offset = found_offset;\n \t}\n \n@@ -1399,8 +1401,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n-\tif (entry->in_pack) {\n-\t\tstruct packed_git *p = entry->in_pack;\n+\tif (IN_PACK(entry)) {\n+\t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1535,14 +1537,16 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tconst struct packed_git *a_in_pack = IN_PACK(a);\n+\tconst struct packed_git *b_in_pack = IN_PACK(b);\n \n \t/* avoid filesystem trashing with loose objects */\n-\tif (!a->in_pack && !b->in_pack)\n+\tif (!a_in_pack && !b_in_pack)\n \t\treturn oidcmp(&a->idx.oid, &b->idx.oid);\n \n-\tif (a->in_pack < b->in_pack)\n+\tif (a_in_pack < b_in_pack)\n \t\treturn -1;\n-\tif (a->in_pack > b->in_pack)\n+\tif (a_in_pack > b_in_pack)\n \t\treturn 1;\n \treturn a->in_pack_offset < b->in_pack_offset ? -1 :\n \t\t\t(a->in_pack_offset > b->in_pack_offset);\n@@ -1578,7 +1582,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \n \toi.sizep = &entry->size;\n \toi.typep = &type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1848,8 +1852,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && trg_entry->in_pack &&\n-\t trg_entry->in_pack == src_entry->in_pack &&\n+\tif (reuse_delta && IN_PACK(trg_entry) &&\n+\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -3193,6 +3197,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tprepare_packing_data(&to_pack);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/cache.h b/cache.h\nindex 862bdff83a..b90feb3802 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1635,6 +1635,7 @@ extern struct packed_git {\n \tint index_version;\n \ttime_t mtime;\n \tint pack_fd;\n+\tint index;\t\t/* for builtin/pack-objects.c */\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n \t\t freshened:1,\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 9558d13834..09f0a88865 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -2,6 +2,8 @@\n #include \"object.h\"\n #include \"pack.h\"\n #include \"pack-objects.h\"\n+#include \"packfile.h\"\n+#include \"config.h\"\n \n static uint32_t locate_object_entry_hash(struct packing_data *pdata,\n \t\t\t\t\t const unsigned char *sha1,\n@@ -86,6 +88,64 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \treturn &pdata->objects[pdata->index[i] - 1];\n }\n \n+static void prepare_in_pack_by_idx(struct packing_data *pdata)\n+{\n+\tstruct packed_git **mapping, *p;\n+\tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n+\n+\tALLOC_ARRAY(mapping, nr);\n+\t/*\n+\t * oe_in_pack() on an all-zero'd object_entry\n+\t * (i.e. in_pack_idx also zero) should return NULL.\n+\t */\n+\tmapping[cnt++] = NULL;\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next, cnt++) {\n+\t\tif (cnt == nr) {\n+\t\t\tfree(mapping);\n+\t\t\treturn;\n+\t\t}\n+\t\tp->index = cnt;\n+\t\tmapping[cnt] = p;\n+\t}\n+\tpdata->in_pack_by_idx = mapping;\n+}\n+\n+/*\n+ * A new pack appears after prepare_in_pack_by_idx() has been\n+ * run. This is likely a race.\n+ *\n+ * We could map this new pack to in_pack_by_idx[] array, but then we\n+ * have to deal with full array anyway. And since it's hard to test\n+ * this fall back code, just stay simple and fall back to using\n+ * in_pack[] array.\n+ */\n+void oe_map_new_pack(struct packing_data *pack,\n+\t\t struct packed_git *p)\n+{\n+\tuint32_t i;\n+\n+\tREALLOC_ARRAY(pack->in_pack, pack->nr_alloc);\n+\n+\tfor (i = 0; i < pack->nr_objects; i++)\n+\t\tpack->in_pack[i] = oe_in_pack(pack, pack->objects + i);\n+\n+\tFREE_AND_NULL(pack->in_pack_by_idx);\n+}\n+\n+/* assume pdata is already zero'd by caller */\n+void prepare_packing_data(struct packing_data *pdata)\n+{\n+\tif (git_env_bool(\"GIT_TEST_FULL_IN_PACK_ARRAY\", 0)) {\n+\t\t/*\n+\t\t * do not initialize in_pack_by_idx[] to force the\n+\t\t * slow path in oe_in_pack()\n+\t\t */\n+\t} else {\n+\t\tprepare_in_pack_by_idx(pdata);\n+\t}\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos)\n@@ -95,6 +155,9 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \tif (pdata->nr_objects >= pdata->nr_alloc) {\n \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\n+\n+\t\tif (!pdata->in_pack_by_idx)\n+\t\t\tREALLOC_ARRAY(pdata->in_pack, pdata->nr_alloc);\n \t}\n \n \tnew_entry = pdata->objects + pdata->nr_objects++;\n@@ -107,5 +170,8 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \telse\n \t\tpdata->index[index_pos] = pdata->nr_objects;\n \n+\tif (pdata->in_pack)\n+\t\tpdata->in_pack[pdata->nr_objects - 1] = NULL;\n+\n \treturn new_entry;\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 71ea992c3c..4a8aa56042 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -3,6 +3,7 @@\n \n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n+#define OE_IN_PACK_BITS\t\t10\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -65,7 +66,7 @@ enum dfs_state {\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\tstruct packed_git *in_pack;\t/* already in pack */\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n \tstruct object_entry *delta_child; /* deltified objects who bases me */\n@@ -100,8 +101,18 @@ struct packing_data {\n \tuint32_t index_size;\n \n \tunsigned int *in_pack_pos;\n+\n+\t/*\n+\t * Only one of these can be non-NULL and they have different\n+\t * sizes. if in_pack_by_idx is allocated, oe_in_pack() returns\n+\t * the pack of an object using in_pack_idx field. If not,\n+\t * in_pack[] array is used the same way as in_pack_pos[]\n+\t */\n+\tstruct packed_git **in_pack_by_idx;\n+\tstruct packed_git **in_pack;\n };\n \n+void prepare_packing_data(struct packing_data *pdata);\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos);\n@@ -158,4 +169,27 @@ static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n \tpack->in_pack_pos[e - pack->objects] = pos;\n }\n \n+static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (pack->in_pack_by_idx)\n+\t\treturn pack->in_pack_by_idx[e->in_pack_idx];\n+\telse\n+\t\treturn pack->in_pack[e - pack->objects];\n+}\n+\n+void oe_map_new_pack(struct packing_data *pack,\n+\t\t struct packed_git *p);\n+static inline void oe_set_in_pack(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tif (!p->index)\n+\t\toe_map_new_pack(pack, p);\n+\tif (pack->in_pack_by_idx)\n+\t\te->in_pack_idx = p->index;\n+\telse\n+\t\tpack->in_pack[e - pack->objects] = p;\n+}\n+\n #endif\ndiff --git a/t/README b/t/README\nindex 09eb2b9768..c6130ff16d 100644\n--- a/t/README\n+++ b/t/README\n@@ -302,6 +302,10 @@ environment set.\n \n GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n \n+GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n+path where there are more than 1024 packs even if the actual number of\n+packs in repository is below this limit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 9c68b99251..5c076637ff 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -457,6 +457,11 @@ test_expect_success !PTHREADS,C_LOCALE_OUTPUT 'pack-objects --threads=N or pack.\n \tgrep -F \"no threads support, ignoring pack.threads\" err\n '\n \n+test_expect_success 'pack-objects in too-many-packs mode' '\n+\tGIT_TEST_FULL_IN_PACK_ARRAY=1 git repack -ad &&\n+\tgit fsck\n+'\n+\n #\n # WARNING!\n #\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343507","messageId":"20180331100311.32373-7-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 06/15] pack-objects: move in_pack_pos out of struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:02Z","receivedAt":"2018-03-31T10:03:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This field is only need for pack-bitmap, which is an optional\nfeature. Move it to a separate array that is only allocated when\npack-bitmap is used (like objects[], it is not freed, since we need it\nuntil the end of the process)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 ++-\n pack-bitmap-write.c | 8 +++++---\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +++-\n pack-objects.h | 16 +++++++++++++++-\n 5 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2ce05626d2..7a672366bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -879,7 +879,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(oid.hash);\n-\t\t\t\tbitmap_writer_build_type_index(written_list, nr_written);\n+\t\t\t\tbitmap_writer_build_type_index(\n+\t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n \t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fd11f08940..f7c897515b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,8 @@ void bitmap_writer_show_progress(int show)\n /**\n * Build the initial type index for the packfile\n */\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr)\n {\n \tuint32_t i;\n@@ -57,12 +58,13 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \twriter.trees = ewah_new();\n \twriter.blobs = ewah_new();\n \twriter.tags = ewah_new();\n+\tALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects);\n \n \tfor (i = 0; i < index_nr; ++i) {\n \t\tstruct object_entry *entry = (struct object_entry *)index[i];\n \t\tenum object_type real_type;\n \n-\t\tentry->in_pack_pos = i;\n+\t\toe_set_in_pack_pos(to_pack, entry, i);\n \n \t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n@@ -147,7 +149,7 @@ static uint32_t find_object_pos(const unsigned char *sha1)\n \t\t\t\"(object %s is missing)\", sha1_to_hex(sha1));\n \t}\n \n-\treturn entry->in_pack_pos;\n+\treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n static void show_object(struct object *object, const char *name, void *data)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9270983e5f..865d9ecc4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1032,7 +1032,7 @@ int rebuild_existing_bitmaps(struct packing_data *mapping,\n \t\toe = packlist_find(mapping, sha1, NULL);\n \n \t\tif (oe)\n-\t\t\treposition[i] = oe->in_pack_pos + 1;\n+\t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n \trebuild = bitmap_new();\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3742a00e14..5ded2f139a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,7 +44,9 @@ int rebuild_existing_bitmaps(struct packing_data *mapping, khash_sha1 *reused_bi\n \n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(unsigned char *sha1);\n-void bitmap_writer_build_type_index(struct pack_idx_entry **index, uint32_t index_nr);\n+void bitmap_writer_build_type_index(struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr);\n void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex cdce1648de..71ea992c3c 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -79,7 +79,6 @@ struct object_entry {\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n@@ -99,6 +98,8 @@ struct packing_data {\n \n \tint32_t *index;\n \tuint32_t index_size;\n+\n+\tunsigned int *in_pack_pos;\n };\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\n@@ -144,4 +145,17 @@ static inline void oe_set_type(struct object_entry *e,\n \te->type_ = (unsigned)type;\n }\n \n+static inline unsigned int oe_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\treturn pack->in_pack_pos[e - pack->objects];\n+}\n+\n+static inline void oe_set_in_pack_pos(const struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e,\n+\t\t\t\t unsigned int pos)\n+{\n+\tpack->in_pack_pos[e - pack->objects] = pos;\n+}\n+\n #endif\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343508","messageId":"20180331100311.32373-6-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 05/15] pack-objects: use bitfield for object_entry::depth","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:01Z","receivedAt":"2018-03-31T10:03:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Because of struct packing from now on we can only handle max depth\n4095 (or even lower when new booleans are added in this struct). This\nshould be ok since long delta chain will cause significant slow down\nanyway.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 1 +\n Documentation/git-pack-objects.txt | 4 +++-\n Documentation/git-repack.txt | 4 +++-\n builtin/pack-objects.c | 6 ++++++\n pack-objects.h | 5 ++---\n 5 files changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f57e9cf10c..9bd3f5a789 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2412,6 +2412,7 @@ pack.window::\n pack.depth::\n \tThe maximum delta depth used by linkgit:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n+\tMaximum value is 4095.\n \n pack.windowMemory::\n \tThe maximum size of memory that is consumed by each thread\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..3503c9e3e6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,7 +96,9 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..25c83c4927 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -90,7 +90,9 @@ other objects in that pack they already have locally.\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for --window is 10 and --depth is 50.\n++\n+The default value for --window is 10 and --depth is 50. The maximum\n+depth is 4095.\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ebb6e034cb..2ce05626d2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,6 +3068,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (depth >= (1 << OE_DEPTH_BITS)) {\n+\t\twarning(_(\"delta chain depth %d is too deep, forcing %d\"),\n+\t\t\tdepth, (1 << OE_DEPTH_BITS) - 1);\n+\t\tdepth = (1 << OE_DEPTH_BITS) - 1;\n+\t}\n+\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 080ef62d31..cdce1648de 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -2,6 +2,7 @@\n #define PACK_OBJECTS_H\n \n #define OE_DFS_STATE_BITS\t2\n+#define OE_DEPTH_BITS\t\t12\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -89,9 +90,7 @@ struct object_entry {\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n-\n-\tint depth;\n-\n+\tunsigned depth:OE_DEPTH_BITS;\n };\n \n struct packing_data {\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343509","messageId":"20180331100311.32373-5-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 04/15] pack-objects: use bitfield for object_entry::dfs_state","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:00Z","receivedAt":"2018-03-31T10:03:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 3 +++\n pack-objects.h | 28 +++++++++++++++++-----------\n 2 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7133baa63f..ebb6e034cb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3049,6 +3049,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n+\t\tBUG(\"too many dfs states, increase OE_DFS_STATE_BITS\");\n+\n \tcheck_replace_refs = 0;\n \n \treset_pack_idx_option(&pack_idx_opts);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b4a83a6123..080ef62d31 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define OE_DFS_STATE_BITS\t2\n+\n+/*\n+ * State flags for depth-first search used for analyzing delta cycles.\n+ *\n+ * The depth is measured in delta-links to the base (so if A is a delta\n+ * against B, then A has a depth of 1, and B a depth of 0).\n+ */\n+enum dfs_state {\n+\tDFS_NONE = 0,\n+\tDFS_ACTIVE,\n+\tDFS_DONE,\n+\tDFS_NUM_STATES\n+};\n+\n /*\n * basic object info\n * -----------------\n@@ -73,19 +88,10 @@ struct object_entry {\n \tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n+\tunsigned dfs_state:OE_DFS_STATE_BITS;\n \n-\t/*\n-\t * State flags for depth-first search used for analyzing delta cycles.\n-\t *\n-\t * The depth is measured in delta-links to the base (so if A is a delta\n-\t * against B, then A has a depth of 1, and B a depth of 0).\n-\t */\n-\tenum {\n-\t\tDFS_NONE = 0,\n-\t\tDFS_ACTIVE,\n-\t\tDFS_DONE\n-\t} dfs_state;\n \tint depth;\n+\n };\n \n struct packing_data {\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343510","messageId":"20180331100311.32373-9-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 08/15] pack-objects: refer to delta objects by index instead of pointer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:04Z","receivedAt":"2018-03-31T10:03:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These delta pointers always point to elements in the objects[] array\nin packing_data struct. We can only hold maximum 4G of those objects\nbecause the array size in nr_objects is uint32_t. We could use\nuint32_t indexes to address these elements instead of pointers. On\n64-bit architecture (8 bytes per pointer) this would save 4 bytes per\npointer.\n\nConvert these delta pointers to indexes. Since we need to handle NULL\npointers as well, the index is shifted by one [1].\n\n[1] This means we can only index 2^32-2 objects even though nr_objects\n could contain 2^32-1 objects. It should not be a problem in\n practice because when we grow objects[], nr_alloc would probably\n blow up long before nr_objects hits the wall.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 116 ++++++++++++++++++++++-------------------\n pack-objects.h | 67 ++++++++++++++++++++++--\n 2 files changed, 124 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4e5812a053..118c8fd993 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,12 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define DELTA(obj) oe_delta(&to_pack, obj)\n+#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n+#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n+#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n+#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -127,11 +133,11 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(entry->delta->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&entry->delta->idx.oid));\n+\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n \tif (!delta_buf || delta_size != entry->delta_size)\n@@ -288,12 +294,12 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tsize = entry->delta_size;\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n \t\tsize = entry->delta_size;\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -317,7 +323,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -343,7 +349,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,8 +385,8 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n \n-\tif (entry->delta)\n-\t\ttype = (allow_ofs_delta && entry->delta->idx.offset) ?\n+\tif (DELTA(entry))\n+\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry->size);\n@@ -408,7 +414,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -427,7 +433,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, entry->delta->idx.oid.hash, 20);\n+\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -467,13 +473,13 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!entry->delta)\n+\tif (!DELTA(entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (entry->delta->idx.offset == (off_t)-1)\n+\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (entry->delta->idx.offset)\n+\telse if (DELTA(entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n@@ -489,7 +495,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (entry->delta)\n+\telse if (DELTA(entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -541,12 +547,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (e->delta) {\n+\tif (DELTA(e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, e->delta, offset)) {\n+\t\tswitch (write_one(f, DELTA(e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\te->delta = NULL;\n+\t\t\tSET_DELTA(e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -608,34 +614,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = e->delta_sibling; s; s = s->delta_sibling) {\n+\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (e->delta_child) {\n+\t\tif (DELTA_CHILD(e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = e->delta_child;\n+\t\t\te = DELTA_CHILD(e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (e->delta_sibling) {\n-\t\t\t\te = e->delta_sibling;\n+\t\t\tif (DELTA_SIBLING(e)) {\n+\t\t\t\te = DELTA_SIBLING(e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = e->delta;\n-\t\t\twhile (e && !e->delta_sibling) {\n+\t\t\te = DELTA(e);\n+\t\t\twhile (e && !DELTA_SIBLING(e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = e->delta;\n+\t\t\t\te = DELTA(e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = e->delta_sibling;\n+\t\t\te = DELTA_SIBLING(e);\n \t\t}\n \t};\n }\n@@ -646,7 +652,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; root->delta; root = root->delta)\n+\tfor (root = e; DELTA(root); root = DELTA(root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -661,8 +667,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tobjects[i].delta_child = NULL;\n-\t\tobjects[i].delta_sibling = NULL;\n+\t\tSET_DELTA_CHILD(&objects[i], NULL);\n+\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n \t}\n \n \t/*\n@@ -672,11 +678,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!e->delta)\n+\t\tif (!DELTA(e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling = e->delta->delta_child;\n-\t\te->delta->delta_child = e;\n+\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n+\t\tSET_DELTA_CHILD(DELTA(e), e);\n \t}\n \n \t/*\n@@ -1493,10 +1499,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->delta = base_entry;\n+\t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n-\t\t\tentry->delta_sibling = base_entry->delta_child;\n-\t\t\tbase_entry->delta_child = entry;\n+\t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n+\t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1567,17 +1573,19 @@ static int pack_offset_sort(const void *_a, const void *_b)\n */\n static void drop_reused_delta(struct object_entry *entry)\n {\n-\tstruct object_entry **p = &entry->delta->delta_child;\n+\tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n \n-\twhile (*p) {\n-\t\tif (*p == entry)\n-\t\t\t*p = (*p)->delta_sibling;\n+\twhile (*idx) {\n+\t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n+\n+\t\tif (oe == entry)\n+\t\t\t*idx = oe->delta_sibling_idx;\n \t\telse\n-\t\t\tp = &(*p)->delta_sibling;\n+\t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tentry->delta = NULL;\n+\tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n@@ -1617,7 +1625,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = cur->delta, total_depth++) {\n+\t cur = DELTA(cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1642,7 +1650,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!cur->delta) {\n+\t\tif (!DELTA(cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1665,7 +1673,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (cur->delta->dfs_state == DFS_ACTIVE) {\n+\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1680,7 +1688,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = cur->delta;\n+\t\tnext = DELTA(cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1865,7 +1873,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n-\tif (!trg_entry->delta) {\n+\tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n@@ -1941,7 +1949,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (!delta_buf)\n \t\treturn 0;\n \n-\tif (trg_entry->delta) {\n+\tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n \t\t src->depth + 1 >= trg->depth) {\n@@ -1970,7 +1978,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\ttrg_entry->delta = src_entry;\n+\tSET_DELTA(trg_entry, src_entry);\n \ttrg_entry->delta_size = delta_size;\n \ttrg->depth = src->depth + 1;\n \n@@ -1979,13 +1987,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = me->delta_child;\n+\tstruct object_entry *child = DELTA_CHILD(me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = child->delta_sibling;\n+\t\tchild = DELTA_SIBLING(child);\n \t}\n \treturn m;\n }\n@@ -2054,7 +2062,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (entry->delta_child) {\n+\t\tif (DELTA_CHILD(entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2104,7 +2112,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && max_depth <= n->depth)\n+\t\tif (DELTA(entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2112,7 +2120,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (entry->delta) {\n+\t\tif (DELTA(entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2433,7 +2441,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (entry->delta)\n+\t\tif (DELTA(entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 4a8aa56042..272ddeeedb 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -68,11 +68,11 @@ struct object_entry {\n \tunsigned long size;\t/* uncompressed size */\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n-\tstruct object_entry *delta;\t/* delta base object */\n-\tstruct object_entry *delta_child; /* deltified objects who bases me */\n-\tstruct object_entry *delta_sibling; /* other deltified objects who\n-\t\t\t\t\t * uses the same base as me\n-\t\t\t\t\t */\n+\tuint32_t delta_idx;\t/* delta base object */\n+\tuint32_t delta_child_idx; /* deltified objects who bases me */\n+\tuint32_t delta_sibling_idx; /* other deltified objects who\n+\t\t\t\t * uses the same base as me\n+\t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n@@ -192,4 +192,61 @@ static inline void oe_set_in_pack(struct packing_data *pack,\n \t\tpack->in_pack[e - pack->objects] = p;\n }\n \n+static inline struct object_entry *oe_delta(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_idx)\n+\t\treturn &pack->objects[e->delta_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta(struct packing_data *pack,\n+\t\t\t\tstruct object_entry *e,\n+\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_child(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_child_idx)\n+\t\treturn &pack->objects[e->delta_child_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_child(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t struct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_child_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_child_idx = 0;\n+}\n+\n+static inline struct object_entry *oe_delta_sibling(\n+\t\tconst struct packing_data *pack,\n+\t\tconst struct object_entry *e)\n+{\n+\tif (e->delta_sibling_idx)\n+\t\treturn &pack->objects[e->delta_sibling_idx - 1];\n+\treturn NULL;\n+}\n+\n+static inline void oe_set_delta_sibling(struct packing_data *pack,\n+\t\t\t\t\tstruct object_entry *e,\n+\t\t\t\t\tstruct object_entry *delta)\n+{\n+\tif (delta)\n+\t\te->delta_sibling_idx = (delta - pack->objects) + 1;\n+\telse\n+\t\te->delta_sibling_idx = 0;\n+}\n+\n #endif\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343511","messageId":"20180331100311.32373-15-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 14/15] pack-objects: reorder members to shrink struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:10Z","receivedAt":"2018-03-31T10:03:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Previous patches leave lots of holes and padding in this struct. This\npatch reorders the members and shrinks the struct down to 80 bytes\n(from 136 bytes on 64-bit systems, before any field shrinking is done)\nwith 16 bits to spare (and a couple more in in_pack_header_size when\nwe really run out of bits).\n\nThis is the last in a series of memory reduction patches (see\n\"pack-objects: a bit of document about struct object_entry\" for the\nfirst one).\n\nOverall they've reduced repack memory size on linux-2.6.git from\n3.747G to 3.424G, or by around 320M, a decrease of 8.5%. The runtime\nof repack has stayed the same throughout this series. Ævar's testing\non a big monorepo he has access to (bigger than linux-2.6.git) has\nshown a 7.9% reduction, so the overall expected improvement should be\nsomewhere around 8%.\n\nSee 87po42cwql.fsf@evledraar.gmail.com on-list\n(https://public-inbox.org/git/87po42cwql.fsf@evledraar.gmail.com/) for\nmore detailed numbers and a test script used to produce the numbers\ncited above.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pack-objects.h | 28 +++++++++++++++++++++-------\n 1 file changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b5114a70a7..60192cce1f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -26,6 +26,10 @@ enum dfs_state {\n };\n \n /*\n+ * The size of struct nearly determines pack-objects's memory\n+ * consumption. This struct is packed tight for that reason. When you\n+ * add or reorder something in this struct, think a bit about this.\n+ *\n * basic object info\n * -----------------\n * idx.oid is filled up before delta searching starts. idx.crc32 is\n@@ -74,34 +78,44 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n+\tvoid *delta_data;\t/* cached delta (uncompressed) */\n+\toff_t in_pack_offset;\n+\tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned size_:OE_SIZE_BITS;\n \tunsigned size_valid:1;\n-\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n-\toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n \tunsigned delta_size_valid:1;\n+\tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n+\tunsigned type_valid:1;\n \tunsigned type_:TYPE_BITS;\n+\tunsigned no_try_delta:1;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n-\tunsigned type_valid:1;\n-\tuint32_t hash;\t\t\t/* name hint hash */\n-\tunsigned char in_pack_header_size;\n \tunsigned preferred_base:1; /*\n \t\t\t\t * we do not pack this, but is available\n \t\t\t\t * to be used as the base object to delta\n \t\t\t\t * objects against.\n \t\t\t\t */\n-\tunsigned no_try_delta:1;\n \tunsigned tagged:1; /* near the very tip of refs */\n \tunsigned filled:1; /* assigned write-order */\n \tunsigned dfs_state:OE_DFS_STATE_BITS;\n+\tunsigned char in_pack_header_size;\n \tunsigned depth:OE_DEPTH_BITS;\n+\n+\t/*\n+\t * pahole results on 64-bit linux (gcc and clang)\n+\t *\n+\t * size: 80, bit_padding: 20 bits, holes: 8 bits\n+\t *\n+\t * and on 32-bit (gcc)\n+\t *\n+\t * size: 76, bit_padding: 20 bits, holes: 8 bits\n+\t */\n };\n \n struct packing_data {\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343512","messageId":"20180331100311.32373-16-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 15/15] ci: exercise the whole test suite with uncommon code in pack-objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:11Z","receivedAt":"2018-03-31T10:03:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Some recent optimizations have been added to pack-objects to reduce\nmemory usage and some code paths are split into two: one for common\nuse cases and one for rare ones. Make sure the rare cases are tested\nwith Travis since it requires manual test configuration that is\nunlikely to be done by developers.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n ci/run-tests.sh | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex 73e273fac7..857d144ee8 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -10,7 +10,10 @@ ln -s \"$cache_dir/.prove\" t/.prove\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\n-\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n+\texport GIT_TEST_SPLIT_INDEX=YesPlease\n+\texport GIT_TEST_FULL_IN_PACK_ARRAY=true\n+\texport GIT_TEST_OE_SIZE=10\n+\tmake --quiet test\n fi\n \n check_unignored_build_artifacts\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343513","messageId":"20180331100311.32373-11-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 10/15] pack-objects: don't check size when the object is bad","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:06Z","receivedAt":"2018-03-31T10:03:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"sha1_object_info() in check_objects() may fail to locate an object in\nthe pack and return type OBJ_BAD. In that case, it will likely leave\nthe \"size\" field untouched. We delay error handling until later in\nprepare_pack() though. Until then, do not touch \"size\" field.\n\nThis field should contain the default value zero, but we can't say\nsha1_object_info() cannot damage it. This becomes more important later\nwhen the object size may have to be retrieved back from the\n(non-existing) pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 34e285a1b7..481b55c746 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1741,7 +1741,7 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (big_file_threshold < entry->size)\n+\t\tif (entry->type_valid && big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -2454,7 +2454,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (entry->size < 50)\n+\t\tif (!entry->type_valid || entry->size < 50)\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343514","messageId":"20180331100311.32373-14-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 13/15] pack-objects: shrink delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:09Z","receivedAt":"2018-03-31T10:03:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Allowing a delta size of 64 bits is crazy. Shrink this field down to\n20 bits with one overflow bit.\n\nIf we find an existing delta larger than 1MB, we do not cache\ndelta_size at all and will get the value from oe_size(), potentially\nfrom disk if it's larger than 4GB.\n\nNote, since DELTA_SIZE() is used in try_delta() code, it must be\nthread-safe. Luckily oe_size() does guarantee this so we it is\nthread-safe.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 26 ++++++++++++++++----------\n pack-objects.h | 23 ++++++++++++++++++++++-\n 2 files changed, 38 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b4ea6290f9..b5bba2c228 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -32,10 +32,12 @@\n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n #define SIZE(obj) oe_size(&to_pack, obj)\n #define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n+#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n #define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n+#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n@@ -142,7 +144,7 @@ static void *get_delta(struct object_entry *entry)\n \t\t oid_to_hex(&DELTA(entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != entry->delta_size)\n+\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -293,14 +295,14 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = entry->delta_size;\n+\t\tsize = DELTA_SIZE(entry);\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n@@ -1508,7 +1510,7 @@ static void check_object(struct object_entry *entry)\n \t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = in_pack_size;\n+\t\t\tSET_DELTA_SIZE(entry, in_pack_size);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1938,7 +1940,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = trg_entry->delta_size;\n+\t\tmax_size = DELTA_SIZE(trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n@@ -2009,10 +2011,14 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n+\tif (delta_size >= (1U << OE_DELTA_SIZE_BITS)) {\n+\t\tfree(delta_buf);\n+\t\treturn 0;\n+\t}\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == trg_entry->delta_size &&\n+\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -2027,7 +2033,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= trg_entry->delta_size;\n+\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -2040,7 +2046,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t}\n \n \tSET_DELTA(trg_entry, src_entry);\n-\ttrg_entry->delta_size = delta_size;\n+\tSET_DELTA_SIZE(trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2163,11 +2169,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n \t\t\tif (size < (1U << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 27697be5c9..b5114a70a7 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -10,6 +10,7 @@\n * above this limit. Don't lower it too much.\n */\n #define OE_SIZE_BITS\t\t31\n+#define OE_DELTA_SIZE_BITS\t20\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -83,7 +84,8 @@ struct object_entry {\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n-\tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tunsigned delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tunsigned delta_size_valid:1;\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n@@ -307,4 +309,23 @@ static inline void oe_set_size(struct packing_data *pack,\n \t}\n }\n \n+static inline unsigned long oe_delta_size(struct packing_data *pack,\n+\t\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->delta_size_valid)\n+\t\treturn e->delta_size_;\n+\treturn oe_size(pack, e);\n+}\n+\n+static inline void oe_set_delta_size(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t unsigned long size)\n+{\n+\te->delta_size_ = size;\n+\te->delta_size_valid = e->delta_size_ == size;\n+\tif (!e->delta_size_valid && size != oe_size(pack, e))\n+\t\tBUG(\"this can only happen in check_object() \"\n+\t\t \"where delta size is the same as entry size\");\n+}\n+\n #endif\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343515","messageId":"20180331100311.32373-10-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 09/15] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:05Z","receivedAt":"2018-03-31T10:03:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only cache deltas when it's smaller than a certain limit. This limit\ndefaults to 1000 but save its compressed length in a 64-bit field.\nShrink that field down to 20 bits, so you can only cache 1MB deltas.\nLarger deltas must be recomputed at when the pack is written down.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 3 ++-\n builtin/pack-objects.c | 24 ++++++++++++++++++------\n pack-objects.h | 3 ++-\n 3 files changed, 22 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9bd3f5a789..00fa824448 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2449,7 +2449,8 @@ pack.deltaCacheLimit::\n \tThe maximum size of a delta, that is cached in\n \tlinkgit:git-pack-objects[1]. This cache is used to speed up the\n \twriting object phase by not having to recompute the final delta\n-\tresult once the best match for all objects is found. Defaults to 1000.\n+\tresult once the best match for all objects is found.\n+\tDefaults to 1000. Maximum value is 65535.\n \n pack.threads::\n \tSpecifies the number of threads to spawn when searching for best\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 118c8fd993..34e285a1b7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2100,12 +2100,19 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * between writes at that moment.\n \t\t */\n \t\tif (entry->delta_data && !pack_to_stdout) {\n-\t\t\tentry->z_delta_size = do_compress(&entry->delta_data,\n-\t\t\t\t\t\t\t entry->delta_size);\n-\t\t\tcache_lock();\n-\t\t\tdelta_cache_size -= entry->delta_size;\n-\t\t\tdelta_cache_size += entry->z_delta_size;\n-\t\t\tcache_unlock();\n+\t\t\tunsigned long size;\n+\n+\t\t\tsize = do_compress(&entry->delta_data, entry->delta_size);\n+\t\t\tif (size < (1U << OE_Z_DELTA_BITS)) {\n+\t\t\t\tentry->z_delta_size = size;\n+\t\t\t\tcache_lock();\n+\t\t\t\tdelta_cache_size -= entry->delta_size;\n+\t\t\t\tdelta_cache_size += entry->z_delta_size;\n+\t\t\t\tcache_unlock();\n+\t\t\t} else {\n+\t\t\t\tFREE_AND_NULL(entry->delta_data);\n+\t\t\t\tentry->z_delta_size = 0;\n+\t\t\t}\n \t\t}\n \n \t\t/* if we made n a delta, and if n is already at max\n@@ -3086,6 +3093,11 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tdepth, (1 << OE_DEPTH_BITS) - 1);\n \t\tdepth = (1 << OE_DEPTH_BITS) - 1;\n \t}\n+\tif (cache_max_small_delta_size >= (1U << OE_Z_DELTA_BITS)) {\n+\t\twarning(_(\"pack.deltaCacheLimit is too high, forcing %d\"),\n+\t\t\t(1U << OE_Z_DELTA_BITS) - 1);\n+\t\tcache_max_small_delta_size = (1U << OE_Z_DELTA_BITS) - 1;\n+\t}\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 272ddeeedb..5613a3d040 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,6 +4,7 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t10\n+#define OE_Z_DELTA_BITS\t\t20\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -75,7 +76,7 @@ struct object_entry {\n \t\t\t\t */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n-\tunsigned long z_delta_size;\t/* delta data size (compressed) */\n+\tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_:TYPE_BITS;\n \tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n \tunsigned type_valid:1;\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343516","messageId":"20180331100311.32373-13-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 12/15] pack-objects: shrink size field in struct object_entry","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:08Z","receivedAt":"2018-03-31T10:04:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's very very rare that an uncompressed object is larger than 4GB\n(partly because Git does not handle those large files very well to\nbegin with). Let's optimize it for the common case where object size\nis smaller than this limit.\n\nShrink size field down to 31 bits and one overflow bit. If the size is\ntoo large, we read it back from disk. As noted in the previous patch,\nwe need to return the delta size instead of canonical size when the\nto-be-reused object entry type is a delta instead of a canonical one.\n\nAdd two compare helpers that can take advantage of the overflow\nbit (e.g. if the file is 4GB+, chances are it's already larger than\ncore.bigFileThreshold and there's no point in comparing the actual\nvalue).\n\nAnother note about oe_get_size_slow(). This function MUST be thread\nsafe because SIZE() macro is used inside try_delta() which may run in\nparallel. Outside parallel code, no-contention locking should be dirt\ncheap (or insignificant compared to i/o access anyway). To exercise\nthis code, it's best to run the test suite with something like\n\n make test GIT_TEST_OE_SIZE=4\n\nwhich forces this code on all objects larger than 3 bytes.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 105 +++++++++++++++++++++++++++++++----------\n pack-objects.c | 3 ++\n pack-objects.h | 57 +++++++++++++++++++++-\n t/README | 6 +++\n 4 files changed, 146 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7a84c3f59a..b4ea6290f9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,8 @@\n #include \"packfile.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n+#define SIZE(obj) oe_size(&to_pack, obj)\n+#define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n #define DELTA(obj) oe_delta(&to_pack, obj)\n #define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n #define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n@@ -274,7 +276,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \n \tif (!usable_delta) {\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n-\t\t entry->size > big_file_threshold &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold) &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n \t\telse {\n@@ -384,12 +386,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n+\tunsigned long entry_size = SIZE(entry);\n \n \tif (DELTA(entry))\n \t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n-\t\t\t\t\t type, entry->size);\n+\t\t\t\t\t type, entry_size);\n \n \toffset = entry->in_pack_offset;\n \trevidx = find_pack_revindex(p, offset);\n@@ -406,7 +409,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tdatalen -= entry->in_pack_header_size;\n \n \tif (!pack_to_stdout && p->index_version == 1 &&\n-\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t check_pack_inflate(p, &w_curs, offset, datalen, entry_size)) {\n \t\terror(\"corrupt packed object for %s\",\n \t\t oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n@@ -1407,6 +1410,8 @@ static void cleanup_preferred_base(void)\n \n static void check_object(struct object_entry *entry)\n {\n+\tunsigned long canonical_size;\n+\n \tif (IN_PACK(entry)) {\n \t\tstruct packed_git *p = IN_PACK(entry);\n \t\tstruct pack_window *w_curs = NULL;\n@@ -1444,7 +1449,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->size = in_pack_size;\n+\t\t\tSET_SIZE(entry, in_pack_size);\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1501,9 +1506,9 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tentry->size = in_pack_size; /* delta size */\n+\t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tentry->delta_size = entry->size;\n+\t\t\tentry->delta_size = in_pack_size;\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n \t\t\tSET_DELTA_CHILD(base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n@@ -1519,9 +1524,10 @@ static void check_object(struct object_entry *entry)\n \t\t\t * object size from the delta header.\n \t\t\t */\n \t\t\tdelta_pos = entry->in_pack_offset + entry->in_pack_header_size;\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs, delta_pos);\n-\t\t\tif (entry->size == 0)\n+\t\t\tcanonical_size = get_size_from_delta(p, &w_curs, delta_pos);\n+\t\t\tif (canonical_size == 0)\n \t\t\t\tgoto give_up;\n+\t\t\tSET_SIZE(entry, canonical_size);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1535,13 +1541,18 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n-\t/*\n-\t * The error condition is checked in prepare_pack(). This is\n-\t * to permit a missing preferred base object to be ignored\n-\t * as a preferred base. Doing so can result in a larger\n-\t * pack file, but the transfer will still take place.\n-\t */\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t &canonical_size));\n+\tif (entry->type_valid) {\n+\t\tSET_SIZE(entry, canonical_size);\n+\t} else {\n+\t\t/*\n+\t\t * Bad object type is checked in prepare_pack(). This is\n+\t\t * to permit a missing preferred base object to be ignored\n+\t\t * as a preferred base. Doing so can result in a larger\n+\t\t * pack file, but the transfer will still take place.\n+\t\t */\n+\t}\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1581,6 +1592,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tunsigned *idx = &to_pack.objects[entry->delta_idx - 1].delta_child_idx;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tenum object_type type;\n+\tunsigned long size;\n \n \twhile (*idx) {\n \t\tstruct object_entry *oe = &to_pack.objects[*idx - 1];\n@@ -1593,7 +1605,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \tSET_DELTA(entry, NULL);\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n+\toi.sizep = &size;\n \toi.typep = &type;\n \tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n@@ -1603,10 +1615,11 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n \t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t\t &entry->size));\n+\t\t\t\t\t\t &size));\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n+\tSET_SIZE(entry, size);\n }\n \n /*\n@@ -1746,7 +1759,8 @@ static void get_object_details(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = sorted_by_offset[i];\n \t\tcheck_object(entry);\n-\t\tif (entry->type_valid && big_file_threshold < entry->size)\n+\t\tif (entry->type_valid &&\n+\t\t oe_size_greater_than(&to_pack, entry, big_file_threshold))\n \t\t\tentry->no_try_delta = 1;\n \t}\n \n@@ -1775,6 +1789,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n+\tunsigned long a_size = SIZE(a);\n+\tunsigned long b_size = SIZE(b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1788,9 +1804,9 @@ static int type_size_sort(const void *_a, const void *_b)\n \t\treturn -1;\n \tif (a->preferred_base < b->preferred_base)\n \t\treturn 1;\n-\tif (a->size > b->size)\n+\tif (a_size > b_size)\n \t\treturn -1;\n-\tif (a->size < b->size)\n+\tif (a_size < b_size)\n \t\treturn 1;\n \treturn a < b ? -1 : (a > b); /* newest first */\n }\n@@ -1843,6 +1859,46 @@ static pthread_mutex_t progress_mutex;\n \n #endif\n \n+/*\n+ * Return the size of the object without doing any delta\n+ * reconstruction (so non-deltas are true object sizes, but deltas\n+ * return the size of the delta data).\n+ */\n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_window *w_curs;\n+\tunsigned char *buf;\n+\tenum object_type type;\n+\tunsigned long used, avail, size;\n+\n+\tif (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {\n+\t\tread_lock();\n+\t\tif (sha1_object_info(e->idx.oid.hash, &size) < 0)\n+\t\t\tdie(_(\"unable to get size of %s\"),\n+\t\t\t oid_to_hex(&e->idx.oid));\n+\t\tread_unlock();\n+\t\treturn size;\n+\t}\n+\n+\tp = oe_in_pack(pack, e);\n+\tif (!p)\n+\t\tBUG(\"when e->type is a delta, it must belong to a pack\");\n+\n+\tread_lock();\n+\tw_curs = NULL;\n+\tbuf = use_pack(p, &w_curs, e->in_pack_offset, &avail);\n+\tused = unpack_object_header_buffer(buf, avail, &type, &size);\n+\tif (used == 0)\n+\t\tdie(_(\"unable to parse object header of %s\"),\n+\t\t oid_to_hex(&e->idx.oid));\n+\n+\tunuse_pack(&w_curs);\n+\tread_unlock();\n+\treturn size;\n+}\n+\n static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t unsigned max_depth, unsigned long *mem_usage)\n {\n@@ -1877,7 +1933,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = trg_entry->size;\n+\ttrg_size = SIZE(trg_entry);\n \tif (!DELTA(trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n@@ -1889,7 +1945,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = src_entry->size;\n+\tsrc_size = SIZE(src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2009,7 +2065,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += n->entry->size;\n+\t\tfreed_mem += SIZE(n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2459,7 +2515,8 @@ static void prepare_pack(int window, int depth)\n \t\t\t */\n \t\t\tcontinue;\n \n-\t\tif (!entry->type_valid || entry->size < 50)\n+\t\tif (!entry->type_valid ||\n+\t\t oe_size_less_than(&to_pack, entry, 50))\n \t\t\tcontinue;\n \n \t\tif (entry->no_try_delta)\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 09f0a88865..bf2e0a808d 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -144,6 +144,9 @@ void prepare_packing_data(struct packing_data *pdata)\n \t} else {\n \t\tprepare_in_pack_by_idx(pdata);\n \t}\n+\n+\tpdata->oe_size_limit = git_env_ulong(\"GIT_TEST_OE_SIZE\",\n+\t\t\t\t\t 1U << OE_SIZE_BITS);\n }\n \n struct object_entry *packlist_alloc(struct packing_data *pdata,\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 534f5a5e4d..27697be5c9 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -5,6 +5,11 @@\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t10\n #define OE_Z_DELTA_BITS\t\t20\n+/*\n+ * Note that oe_set_size() becomes expensive when the given size is\n+ * above this limit. Don't lower it too much.\n+ */\n+#define OE_SIZE_BITS\t\t31\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -68,7 +73,8 @@ enum dfs_state {\n */\n struct object_entry {\n \tstruct pack_idx_entry idx;\n-\tunsigned long size;\t/* uncompressed size */\n+\tunsigned size_:OE_SIZE_BITS;\n+\tunsigned size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \toff_t in_pack_offset;\n \tuint32_t delta_idx;\t/* delta base object */\n@@ -113,6 +119,8 @@ struct packing_data {\n \t */\n \tstruct packed_git **in_pack_by_idx;\n \tstruct packed_git **in_pack;\n+\n+\tuintmax_t oe_size_limit;\n };\n \n void prepare_packing_data(struct packing_data *pdata);\n@@ -252,4 +260,51 @@ static inline void oe_set_delta_sibling(struct packing_data *pack,\n \t\te->delta_sibling_idx = 0;\n }\n \n+unsigned long oe_get_size_slow(struct packing_data *pack,\n+\t\t\t const struct object_entry *e);\n+static inline unsigned long oe_size(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *e)\n+{\n+\tif (e->size_valid)\n+\t\treturn e->size_;\n+\n+\treturn oe_get_size_slow(pack, e);\n+}\n+\n+static inline int oe_size_less_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n+{\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ < rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n+\t\treturn 0;\n+\treturn oe_get_size_slow(pack, lhs) < rhs;\n+}\n+\n+static inline int oe_size_greater_than(struct packing_data *pack,\n+\t\t\t\t const struct object_entry *lhs,\n+\t\t\t\t unsigned long rhs)\n+{\n+\tif (lhs->size_valid)\n+\t\treturn lhs->size_ > rhs;\n+\tif (rhs < pack->oe_size_limit) /* rhs < 2^x <= lhs ? */\n+\t\treturn 1;\n+\treturn oe_get_size_slow(pack, lhs) > rhs;\n+}\n+\n+static inline void oe_set_size(struct packing_data *pack,\n+\t\t\t struct object_entry *e,\n+\t\t\t unsigned long size)\n+{\n+\tif (size < pack->oe_size_limit) {\n+\t\te->size_ = size;\n+\t\te->size_valid = 1;\n+\t} else {\n+\t\te->size_valid = 0;\n+\t\tif (oe_get_size_slow(pack, e) != size)\n+\t\t\tBUG(\"'size' is supposed to be the object size!\");\n+\t}\n+}\n+\n #endif\ndiff --git a/t/README b/t/README\nindex c6130ff16d..c01d210c15 100644\n--- a/t/README\n+++ b/t/README\n@@ -306,6 +306,12 @@ GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n path where there are more than 1024 packs even if the actual number of\n packs in repository is below this limit.\n \n+GIT_TEST_OE_SIZE= exercises the uncommon pack-objects code path\n+where we do not cache object size in memory and read it from existing\n+packs on demand. This normally only happens when the object size is\n+over 2GB. This variable forces the code path on any object larger than\n+ bytes.\n+\n Naming Tests\n ------------\n \n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343517","messageId":"20180331100311.32373-12-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 11/15] pack-objects: clarify the use of object_entry::size","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:03:07Z","receivedAt":"2018-03-31T10:04:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"While this field most of the time contains the canonical object size,\nthere is one case it does not: when we have found that the base object\nof the delta in question is also to be packed, we will very happily\nreuse the delta by copying it over instead of regenerating the new\ndelta.\n\n\"size\" in this case will record the delta size, not canonical object\nsize. Later on in write_reuse_object(), we reconstruct the delta\nheader and \"size\" is used for this purpose. When this happens, the\n\"type\" field contains a delta type instead of a canonical type.\nHighlight this in the code since it could be tricky to see.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 11 ++++++++---\n pack-objects.h | 4 +++-\n 2 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 481b55c746..7a84c3f59a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1417,6 +1417,7 @@ static void check_object(struct object_entry *entry)\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n \t\tenum object_type type;\n+\t\tunsigned long in_pack_size;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1426,7 +1427,7 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n \t\t\t\t\t\t &type,\n-\t\t\t\t\t\t &entry->size);\n+\t\t\t\t\t\t &in_pack_size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n@@ -1443,6 +1444,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tentry->size = in_pack_size;\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1499,6 +1501,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n+\t\t\tentry->size = in_pack_size; /* delta size */\n \t\t\tSET_DELTA(entry, base_entry);\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n@@ -1508,13 +1511,15 @@ static void check_object(struct object_entry *entry)\n \t\t}\n \n \t\tif (oe_type(entry)) {\n+\t\t\toff_t delta_pos;\n+\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n \t\t\t * object size from the delta header.\n \t\t\t */\n-\t\t\tentry->size = get_size_from_delta(p, &w_curs,\n-\t\t\t\t\tentry->in_pack_offset + entry->in_pack_header_size);\n+\t\t\tdelta_pos = entry->in_pack_offset + entry->in_pack_header_size;\n+\t\t\tentry->size = get_size_from_delta(p, &w_curs, delta_pos);\n \t\t\tif (entry->size == 0)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 5613a3d040..534f5a5e4d 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -30,7 +30,9 @@ enum dfs_state {\n *\n * \"size\" is the uncompressed object size. Compressed size of the raw\n * data for an object in a pack is not stored anywhere but is computed\n- * and made available when reverse .idx is made.\n+ * and made available when reverse .idx is made. Note that when a\n+ * delta is reused, \"size\" is the uncompressed _delta_ size, not the\n+ * canonical one after the delta has been applied.\n *\n * \"hash\" contains a path name hash which is used for sorting the\n * delta list and also during delta searching. Once prepare_pack()\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343518","messageId":"20180331100311.32373-4-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"[PATCH v8 03/15] pack-objects: turn type and in_pack_type to bitfields","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:02:59Z","receivedAt":"2018-03-31T10:04:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"An extra field type_valid is added to carry the equivalent of OBJ_BAD\nin the original \"type\" field. in_pack_type always contains a valid\ntype so we only need 3 bits for it.\n\nA note about accepting OBJ_NONE as \"valid\" type. The function\nread_object_list_from_stdin() can pass this value [1] and it\neventually calls create_object_entry() where current code skip setting\n\"type\" field if the incoming type is zero. This does not have any bad\nside effects because \"type\" field should be memset()'d anyway.\n\nBut since we also need to set type_valid now, skipping oe_set_type()\nleaves type_valid zero/false, which will make oe_type() return\nOBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in\nprepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger\n\n fatal: unable to get type of object ...\n\nAccepting OBJ_NONE [2] does sound wrong, but this is how it is has\nbeen for a very long time and I haven't time to dig in further.\n\n[1] See 5c49c11686 (pack-objects: better check_object() performances -\n 2007-04-16)\n\n[2] 21666f1aae (convert object type handling from a string to a number\n - 2007-02-26)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 60 ++++++++++++++++++++++++------------------\n cache.h | 2 ++\n object.h | 1 -\n pack-bitmap-write.c | 6 ++---\n pack-objects.h | 20 ++++++++++++--\n 5 files changed, 58 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c674b2843..7133baa63f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -265,7 +265,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tif (entry->type == OBJ_BLOB &&\n+\t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.oid.hash, &type, &size, NULL)) != NULL)\n \t\t\tbuf = NULL;\n@@ -371,7 +371,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n-\tenum object_type type = entry->type;\n+\tenum object_type type = oe_type(entry);\n \toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n@@ -480,11 +480,12 @@ static off_t write_object(struct hashfile *f,\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n-\telse if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)\n+\telse if (oe_type(entry) == OBJ_REF_DELTA ||\n+\t\t oe_type(entry) == OBJ_OFS_DELTA)\n \t\t\t\t/* check_object() decided it for us ... */\n \t\tto_reuse = usable_delta;\n \t\t\t\t/* ... but pack split may override that */\n-\telse if (entry->type != entry->in_pack_type)\n+\telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n \telse if (entry->delta)\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n@@ -705,8 +706,8 @@ static struct object_entry **compute_write_order(void)\n \t * And then all remaining commits and tags.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_COMMIT &&\n-\t\t objects[i].type != OBJ_TAG)\n+\t\tif (oe_type(&objects[i]) != OBJ_COMMIT &&\n+\t\t oe_type(&objects[i]) != OBJ_TAG)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -715,7 +716,7 @@ static struct object_entry **compute_write_order(void)\n \t * And then all the trees.\n \t */\n \tfor (i = last_untagged; i < to_pack.nr_objects; i++) {\n-\t\tif (objects[i].type != OBJ_TREE)\n+\t\tif (oe_type(&objects[i]) != OBJ_TREE)\n \t\t\tcontinue;\n \t\tadd_to_write_order(wo, &wo_end, &objects[i]);\n \t}\n@@ -1066,8 +1067,7 @@ static void create_object_entry(const struct object_id *oid,\n \n \tentry = packlist_alloc(&to_pack, oid->hash, index_pos);\n \tentry->hash = hash;\n-\tif (type)\n-\t\tentry->type = type;\n+\toe_set_type(entry, type);\n \tif (exclude)\n \t\tentry->preferred_base = 1;\n \telse\n@@ -1407,6 +1407,7 @@ static void check_object(struct object_entry *entry)\n \t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n+\t\tenum object_type type;\n \n \t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);\n \n@@ -1415,11 +1416,15 @@ static void check_object(struct object_entry *entry)\n \t\t * since non-delta representations could still be reused.\n \t\t */\n \t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t &type,\n \t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n+\t\tif (type < 0)\n+\t\t\tBUG(\"invalid type %d\", type);\n+\t\tentry->in_pack_type = type;\n+\n \t\t/*\n \t\t * Determine if this is a delta and if so whether we can\n \t\t * reuse it or not. Otherwise let's find out as cheaply as\n@@ -1428,9 +1433,9 @@ static void check_object(struct object_entry *entry)\n \t\tswitch (entry->in_pack_type) {\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->in_pack_header_size = used;\n-\t\t\tif (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)\n+\t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n@@ -1484,7 +1489,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n \t\t\t */\n-\t\t\tentry->type = entry->in_pack_type;\n+\t\t\toe_set_type(entry, entry->in_pack_type);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -1493,7 +1498,7 @@ static void check_object(struct object_entry *entry)\n \t\t\treturn;\n \t\t}\n \n-\t\tif (entry->type) {\n+\t\tif (oe_type(entry)) {\n \t\t\t/*\n \t\t\t * This must be a delta and we already know what the\n \t\t\t * final object type is. Let's extract the actual\n@@ -1516,7 +1521,7 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->idx.oid.hash, &entry->size);\n+\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash, &entry->size));\n \t/*\n \t * The error condition is checked in prepare_pack(). This is\n \t * to permit a missing preferred base object to be ignored\n@@ -1559,6 +1564,7 @@ static void drop_reused_delta(struct object_entry *entry)\n {\n \tstruct object_entry **p = &entry->delta->delta_child;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum object_type type;\n \n \twhile (*p) {\n \t\tif (*p == entry)\n@@ -1570,16 +1576,18 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->depth = 0;\n \n \toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n+\toi.typep = &type;\n \tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n-\t\t * And if that fails, the error will be recorded in entry->type\n+\t\t * And if that fails, the error will be recorded in oe_type(entry)\n \t\t * and dealt with in prepare_pack().\n \t\t */\n-\t\tentry->type = sha1_object_info(entry->idx.oid.hash,\n-\t\t\t\t\t &entry->size);\n+\t\toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n+\t\t\t\t\t\t &entry->size));\n+\t} else {\n+\t\toe_set_type(entry, type);\n \t}\n }\n \n@@ -1747,10 +1755,12 @@ static int type_size_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n+\tenum object_type a_type = oe_type(a);\n+\tenum object_type b_type = oe_type(b);\n \n-\tif (a->type > b->type)\n+\tif (a_type > b_type)\n \t\treturn -1;\n-\tif (a->type < b->type)\n+\tif (a_type < b_type)\n \t\treturn 1;\n \tif (a->hash > b->hash)\n \t\treturn -1;\n@@ -1826,7 +1836,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tvoid *delta_buf;\n \n \t/* Don't bother doing diffs between different types */\n-\tif (trg_entry->type != src_entry->type)\n+\tif (oe_type(trg_entry) != oe_type(src_entry))\n \t\treturn -1;\n \n \t/*\n@@ -2432,11 +2442,11 @@ static void prepare_pack(int window, int depth)\n \n \t\tif (!entry->preferred_base) {\n \t\t\tnr_deltas++;\n-\t\t\tif (entry->type < 0)\n+\t\t\tif (oe_type(entry) < 0)\n \t\t\t\tdie(\"unable to get type of object %s\",\n \t\t\t\t oid_to_hex(&entry->idx.oid));\n \t\t} else {\n-\t\t\tif (entry->type < 0) {\n+\t\t\tif (oe_type(entry) < 0) {\n \t\t\t\t/*\n \t\t\t\t * This object is not found, but we\n \t\t\t\t * don't have to include it anyway.\n@@ -2545,7 +2555,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected object ID, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, 0, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 21fbcc2414..862bdff83a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -373,6 +373,8 @@ extern void free_name_hash(struct index_state *istate);\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n #endif\n \n+#define TYPE_BITS 3\n+\n enum object_type {\n \tOBJ_BAD = -1,\n \tOBJ_NONE = 0,\ndiff --git a/object.h b/object.h\nindex 87563d9056..8ce294d6ec 100644\n--- a/object.h\n+++ b/object.h\n@@ -25,7 +25,6 @@ struct object_array {\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-#define TYPE_BITS 3\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e01f992884..fd11f08940 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \n \t\tentry->in_pack_pos = i;\n \n-\t\tswitch (entry->type) {\n+\t\tswitch (oe_type(entry)) {\n \t\tcase OBJ_COMMIT:\n \t\tcase OBJ_TREE:\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n-\t\t\treal_type = entry->type;\n+\t\t\treal_type = oe_type(entry);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -98,7 +98,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,\n \t\tdefault:\n \t\t\tdie(\"Missing type information for %s (%d/%d)\",\n \t\t\t oid_to_hex(&entry->idx.oid), real_type,\n-\t\t\t entry->type);\n+\t\t\t oe_type(entry));\n \t\t}\n \t}\n }\ndiff --git a/pack-objects.h b/pack-objects.h\nindex c0a1f61aac..b4a83a6123 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -59,8 +59,9 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n \tunsigned long z_delta_size;\t/* delta data size (compressed) */\n-\tenum object_type type;\n-\tenum object_type in_pack_type;\t/* could be delta */\n+\tunsigned type_:TYPE_BITS;\n+\tunsigned in_pack_type:TYPE_BITS; /* could be delta */\n+\tunsigned type_valid:1;\n \tuint32_t hash;\t\t\t/* name hint hash */\n \tunsigned int in_pack_pos;\n \tunsigned char in_pack_header_size;\n@@ -123,4 +124,19 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline enum object_type oe_type(const struct object_entry *e)\n+{\n+\treturn e->type_valid ? e->type_ : OBJ_BAD;\n+}\n+\n+static inline void oe_set_type(struct object_entry *e,\n+\t\t\t enum object_type type)\n+{\n+\tif (type >= OBJ_ANY)\n+\t\tBUG(\"OBJ_ANY cannot be set in pack-objects code\");\n+\n+\te->type_valid = type >= OBJ_NONE;\n+\te->type_ = (unsigned)type;\n+}\n+\n #endif\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343519","messageId":"20180331100311.32373-1-pclouds@gmail.com","threadId":"47943","inReplyTo":"20180324063353.24722-1-pclouds@gmail.com","subject":"[PATCH v8 00/15] nd/pack-objects-pack-struct updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:02:56Z","receivedAt":"2018-03-31T10:04:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v8 changes\n\n- prefer BUG() over die()\n- do \"1U <<\" instead of \"1 << \" to avoid undefined behavior with\n signed shifting.\n- add more comments based on Jeff's feedback\n- plug a leak in try_delta() when delta_size is too large\n- be kind and set depth/cache_max_small_delta_size to max limit\n instead of dying when the user gives a value over limit\n- make travis execute pack-objects uncommon code\n- use git_env_*() instead of manually handling getenv() values\n- fallback code for when a new pack is added when pack-objects is\n running\n- Compressed cached delta size limit is increased from 64k to 1MB\n- Cached delta size limit is decreased from 2G to 1MB\n\nInterdiff\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c774821930..b5bba2c228 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1439,7 +1439,7 @@ static void check_object(struct object_entry *entry)\n \t\t\tgoto give_up;\n \n \t\tif (type < 0)\n-\t\t\tdie(\"BUG: invalid type %d\", type);\n+\t\t\tBUG(\"invalid type %d\", type);\n \t\tentry->in_pack_type = type;\n \n \t\t/*\n@@ -1861,6 +1861,11 @@ static pthread_mutex_t progress_mutex;\n \n #endif\n \n+/*\n+ * Return the size of the object without doing any delta\n+ * reconstruction (so non-deltas are true object sizes, but deltas\n+ * return the size of the delta data).\n+ */\n unsigned long oe_get_size_slow(struct packing_data *pack,\n \t\t\t const struct object_entry *e)\n {\n@@ -1881,7 +1886,7 @@ unsigned long oe_get_size_slow(struct packing_data *pack,\n \n \tp = oe_in_pack(pack, e);\n \tif (!p)\n-\t\tdie(\"BUG: when e->type is a delta, it must belong to a pack\");\n+\t\tBUG(\"when e->type is a delta, it must belong to a pack\");\n \n \tread_lock();\n \tw_curs = NULL;\n@@ -2006,8 +2011,10 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n \tif (!delta_buf)\n \t\treturn 0;\n-\tif (delta_size >= (1 << OE_DELTA_SIZE_BITS))\n+\tif (delta_size >= (1U << OE_DELTA_SIZE_BITS)) {\n+\t\tfree(delta_buf);\n \t\treturn 0;\n+\t}\n \n \tif (DELTA(trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n@@ -2163,7 +2170,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t\tunsigned long size;\n \n \t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n-\t\t\tif (size < (1 << OE_Z_DELTA_BITS)) {\n+\t\t\tif (size < (1U << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n \t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n@@ -3131,7 +3138,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t};\n \n \tif (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))\n-\t\tdie(\"BUG: too many dfs states, increase OE_DFS_STATE_BITS\");\n+\t\tBUG(\"too many dfs states, increase OE_DFS_STATE_BITS\");\n \n \tcheck_replace_refs = 0;\n \n@@ -3149,12 +3156,16 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n-\tif (depth >= (1 << OE_DEPTH_BITS))\n-\t\tdie(_(\"delta chain depth %d is greater than maximum limit %d\"),\n-\t\t depth, (1 << OE_DEPTH_BITS) - 1);\n-\tif (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))\n-\t\tdie(_(\"pack.deltaCacheLimit is greater than maximum limit %d\"),\n-\t\t (1 << OE_Z_DELTA_BITS) - 1);\n+\tif (depth >= (1 << OE_DEPTH_BITS)) {\n+\t\twarning(_(\"delta chain depth %d is too deep, forcing %d\"),\n+\t\t\tdepth, (1 << OE_DEPTH_BITS) - 1);\n+\t\tdepth = (1 << OE_DEPTH_BITS) - 1;\n+\t}\n+\tif (cache_max_small_delta_size >= (1U << OE_Z_DELTA_BITS)) {\n+\t\twarning(_(\"pack.deltaCacheLimit is too high, forcing %d\"),\n+\t\t\t(1U << OE_Z_DELTA_BITS) - 1);\n+\t\tcache_max_small_delta_size = (1U << OE_Z_DELTA_BITS) - 1;\n+\t}\n \n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n@@ -3274,6 +3285,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tprepare_packing_data(&to_pack);\n+\n \tif (progress)\n \t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n \tif (!use_internal_rev_list)\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex 73e273fac7..857d144ee8 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -10,7 +10,10 @@ ln -s \"$cache_dir/.prove\" t/.prove\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\n-\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n+\texport GIT_TEST_SPLIT_INDEX=YesPlease\n+\texport GIT_TEST_FULL_IN_PACK_ARRAY=true\n+\texport GIT_TEST_OE_SIZE=10\n+\tmake --quiet test\n fi\n \n check_unignored_build_artifacts\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 59c6e40a02..bf2e0a808d 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -3,6 +3,7 @@\n #include \"pack.h\"\n #include \"pack-objects.h\"\n #include \"packfile.h\"\n+#include \"config.h\"\n \n static uint32_t locate_object_entry_hash(struct packing_data *pdata,\n \t\t\t\t\t const unsigned char *sha1,\n@@ -90,18 +91,14 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n static void prepare_in_pack_by_idx(struct packing_data *pdata)\n {\n \tstruct packed_git **mapping, *p;\n-\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n-\n-\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n-\t\t/*\n-\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n-\t\t * used instead\n-\t\t */\n-\t\treturn;\n-\t}\n+\tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n \n \tALLOC_ARRAY(mapping, nr);\n-\tmapping[cnt++] = NULL; /* zero index must be mapped to NULL */\n+\t/*\n+\t * oe_in_pack() on an all-zero'd object_entry\n+\t * (i.e. in_pack_idx also zero) should return NULL.\n+\t */\n+\tmapping[cnt++] = NULL;\n \tprepare_packed_git();\n \tfor (p = packed_git; p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n@@ -114,21 +111,50 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \tpdata->in_pack_by_idx = mapping;\n }\n \n+/*\n+ * A new pack appears after prepare_in_pack_by_idx() has been\n+ * run. This is likely a race.\n+ *\n+ * We could map this new pack to in_pack_by_idx[] array, but then we\n+ * have to deal with full array anyway. And since it's hard to test\n+ * this fall back code, just stay simple and fall back to using\n+ * in_pack[] array.\n+ */\n+void oe_map_new_pack(struct packing_data *pack,\n+\t\t struct packed_git *p)\n+{\n+\tuint32_t i;\n+\n+\tREALLOC_ARRAY(pack->in_pack, pack->nr_alloc);\n+\n+\tfor (i = 0; i < pack->nr_objects; i++)\n+\t\tpack->in_pack[i] = oe_in_pack(pack, pack->objects + i);\n+\n+\tFREE_AND_NULL(pack->in_pack_by_idx);\n+}\n+\n+/* assume pdata is already zero'd by caller */\n+void prepare_packing_data(struct packing_data *pdata)\n+{\n+\tif (git_env_bool(\"GIT_TEST_FULL_IN_PACK_ARRAY\", 0)) {\n+\t\t/*\n+\t\t * do not initialize in_pack_by_idx[] to force the\n+\t\t * slow path in oe_in_pack()\n+\t\t */\n+\t} else {\n+\t\tprepare_in_pack_by_idx(pdata);\n+\t}\n+\n+\tpdata->oe_size_limit = git_env_ulong(\"GIT_TEST_OE_SIZE\",\n+\t\t\t\t\t 1U << OE_SIZE_BITS);\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos)\n {\n \tstruct object_entry *new_entry;\n \n-\tif (!pdata->nr_objects) {\n-\t\tprepare_in_pack_by_idx(pdata);\n-\t\tif (getenv(\"GIT_TEST_OE_SIZE_BITS\")) {\n-\t\t\tint bits = atoi(getenv(\"GIT_TEST_OE_SIZE_BITS\"));;\n-\t\t\tpdata->oe_size_limit = 1 << bits;\n-\t\t}\n-\t\tif (!pdata->oe_size_limit)\n-\t\t\tpdata->oe_size_limit = 1 << OE_SIZE_BITS;\n-\t}\n \tif (pdata->nr_objects >= pdata->nr_alloc) {\n \t\tpdata->nr_alloc = (pdata->nr_alloc + 1024) * 3 / 2;\n \t\tREALLOC_ARRAY(pdata->objects, pdata->nr_alloc);\ndiff --git a/pack-objects.h b/pack-objects.h\nindex c20f67e25b..60192cce1f 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -4,9 +4,13 @@\n #define OE_DFS_STATE_BITS\t2\n #define OE_DEPTH_BITS\t\t12\n #define OE_IN_PACK_BITS\t\t10\n-#define OE_Z_DELTA_BITS\t\t16\n+#define OE_Z_DELTA_BITS\t\t20\n+/*\n+ * Note that oe_set_size() becomes expensive when the given size is\n+ * above this limit. Don't lower it too much.\n+ */\n #define OE_SIZE_BITS\t\t31\n-#define OE_DELTA_SIZE_BITS\t31\n+#define OE_DELTA_SIZE_BITS\t20\n \n /*\n * State flags for depth-first search used for analyzing delta cycles.\n@@ -36,7 +40,7 @@ enum dfs_state {\n *\n * \"size\" is the uncompressed object size. Compressed size of the raw\n * data for an object in a pack is not stored anywhere but is computed\n- * and made available when reverse .idx is made. Note that when an\n+ * and made available when reverse .idx is made. Note that when a\n * delta is reused, \"size\" is the uncompressed _delta_ size, not the\n * canonical one after the delta has been applied.\n *\n@@ -77,15 +81,15 @@ struct object_entry {\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \toff_t in_pack_offset;\n \tuint32_t hash;\t\t\t/* name hint hash */\n-\tuint32_t size_:OE_SIZE_BITS;\n+\tunsigned size_:OE_SIZE_BITS;\n \tunsigned size_valid:1;\n \tuint32_t delta_idx;\t/* delta base object */\n \tuint32_t delta_child_idx; /* deltified objects who bases me */\n \tuint32_t delta_sibling_idx; /* other deltified objects who\n \t\t\t\t * uses the same base as me\n \t\t\t\t */\n-\tuint32_t delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n-\tuint32_t delta_size_valid:1;\n+\tunsigned delta_size_:OE_DELTA_SIZE_BITS; /* delta data size (uncompressed) */\n+\tunsigned delta_size_valid:1;\n \tunsigned in_pack_idx:OE_IN_PACK_BITS;\t/* already in pack */\n \tunsigned z_delta_size:OE_Z_DELTA_BITS;\n \tunsigned type_valid:1;\n@@ -103,7 +107,15 @@ struct object_entry {\n \tunsigned char in_pack_header_size;\n \tunsigned depth:OE_DEPTH_BITS;\n \n-\t/* size: 80, bit_padding: 20 bits, holes: 1 bit */\n+\t/*\n+\t * pahole results on 64-bit linux (gcc and clang)\n+\t *\n+\t * size: 80, bit_padding: 20 bits, holes: 8 bits\n+\t *\n+\t * and on 32-bit (gcc)\n+\t *\n+\t * size: 76, bit_padding: 20 bits, holes: 8 bits\n+\t */\n };\n \n struct packing_data {\n@@ -127,6 +139,7 @@ struct packing_data {\n \tuintmax_t oe_size_limit;\n };\n \n+void prepare_packing_data(struct packing_data *pdata);\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t uint32_t index_pos);\n@@ -164,7 +177,7 @@ static inline void oe_set_type(struct object_entry *e,\n \t\t\t enum object_type type)\n {\n \tif (type >= OBJ_ANY)\n-\t\tdie(\"BUG: OBJ_ANY cannot be set in pack-objects code\");\n+\t\tBUG(\"OBJ_ANY cannot be set in pack-objects code\");\n \n \te->type_valid = type >= OBJ_NONE;\n \te->type_ = (unsigned)type;\n@@ -190,21 +203,20 @@ static inline struct packed_git *oe_in_pack(const struct packing_data *pack,\n \t\treturn pack->in_pack_by_idx[e->in_pack_idx];\n \telse\n \t\treturn pack->in_pack[e - pack->objects];\n-\n }\n \n+void oe_map_new_pack(struct packing_data *pack,\n+\t\t struct packed_git *p);\n static inline void oe_set_in_pack(struct packing_data *pack,\n \t\t\t\t struct object_entry *e,\n \t\t\t\t struct packed_git *p)\n {\n-\tif (pack->in_pack_by_idx) {\n-\t\tif (p->index <= 0)\n-\t\t\tdie(\"BUG: found_pack should be NULL \"\n-\t\t\t\t\t\"instead of having non-positive index\");\n+\tif (!p->index)\n+\t\toe_map_new_pack(pack, p);\n+\tif (pack->in_pack_by_idx)\n \t\te->in_pack_idx = p->index;\n-\t} else\n+\telse\n \t\tpack->in_pack[e - pack->objects] = p;\n-\n }\n \n static inline struct object_entry *oe_delta(\n@@ -307,7 +319,7 @@ static inline void oe_set_size(struct packing_data *pack,\n \t} else {\n \t\te->size_valid = 0;\n \t\tif (oe_get_size_slow(pack, e) != size)\n-\t\t\tdie(\"BUG: 'size' is supposed to be the object size!\");\n+\t\t\tBUG(\"'size' is supposed to be the object size!\");\n \t}\n }\n \n@@ -326,7 +338,7 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n \te->delta_size_ = size;\n \te->delta_size_valid = e->delta_size_ == size;\n \tif (!e->delta_size_valid && size != oe_size(pack, e))\n-\t\tdie(\"BUG: this can only happen in check_object() \"\n+\t\tBUG(\"this can only happen in check_object() \"\n \t\t \"where delta size is the same as entry size\");\n }\n \ndiff --git a/t/README b/t/README\nindex 02bfb3fed5..c01d210c15 100644\n--- a/t/README\n+++ b/t/README\n@@ -291,16 +291,26 @@ expect the rest to function correctly.\n and know what setup is needed for it. Or when you want to run\n everything up to a certain test.\n \n+\n+Running tests with special setups\n+---------------------------------\n+\n+The whole test suite could be run to test some special features\n+that cannot be easily covered by a few specific test cases. These\n+could be enabled by running the test suite with correct GIT_TEST_\n+environment set.\n+\n+GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n+\n GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n path where there are more than 1024 packs even if the actual number of\n packs in repository is below this limit.\n \n-GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n-code path where we do not cache objecct size in memory and read it\n-from existing packs on demand. This normally only happens when the\n-object size is over 2GB. This variable forces the code path on any\n-object larger than 2^ bytes.\n-\n+GIT_TEST_OE_SIZE= exercises the uncommon pack-objects code path\n+where we do not cache object size in memory and read it from existing\n+packs on demand. This normally only happens when the object size is\n+over 2GB. This variable forces the code path on any object larger than\n+ bytes.\n \n Naming Tests\n ------------\nNguyễn Thái Ngọc Duy (15):\n t/README: mention about running the test suite in special modes\n pack-objects: a bit of document about struct object_entry\n pack-objects: turn type and in_pack_type to bitfields\n pack-objects: use bitfield for object_entry::dfs_state\n pack-objects: use bitfield for object_entry::depth\n pack-objects: move in_pack_pos out of struct object_entry\n pack-objects: move in_pack out of struct object_entry\n pack-objects: refer to delta objects by index instead of pointer\n pack-objects: shrink z_delta_size field in struct object_entry\n pack-objects: don't check size when the object is bad\n pack-objects: clarify the use of object_entry::size\n pack-objects: shrink size field in struct object_entry\n pack-objects: shrink delta_size field in struct object_entry\n pack-objects: reorder members to shrink struct object_entry\n ci: exercise the whole test suite with uncommon code in pack-objects\n\n Documentation/config.txt | 4 +-\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 4 +-\n builtin/pack-objects.c | 366 +++++++++++++++++++----------\n cache.h | 3 +\n ci/run-tests.sh | 5 +-\n object.h | 1 -\n pack-bitmap-write.c | 14 +-\n pack-bitmap.c | 2 +-\n pack-bitmap.h | 4 +-\n pack-objects.c | 69 ++++++\n pack-objects.h | 312 ++++++++++++++++++++++--\n t/README | 20 ++\n t/t5300-pack-object.sh | 5 +\n 14 files changed, 650 insertions(+), 163 deletions(-)\n\n-- \n2.17.0.rc2.515.g4feb9b7923\n\n"},{"id":"343520","messageId":"20180331101347.GA32290@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8AQ7yRK65rYqMkR+QJn2KUJ=Y+9E_=k3aBU78xF=LBQjw@mail.gmail.com","subject":"Re: [PATCH v7 10/13] pack-objects: clarify the use of object_entry::size","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-31T10:13:47Z","receivedAt":"2018-03-31T10:13:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 31, 2018 at 06:35:40AM +0200, Duy Nguyen wrote:\n\n> On Fri, Mar 30, 2018 at 11:04 PM, Jeff King wrote:\n> > The subject says \"clarify\" so I was a little surprised to see code\n> > changes. It looks like we're just avoiding reassigning on top of the\n> > value repeatedly, which is part of that clarification. It looks like a\n> > noop to me.\n> \n> Oh well... I was counting on the new name (in_pack_size, which follows\n> in_pack_type naming convention) to emphasize it (and the new \"delta\n> size\" comment to point out where in_pack_size contains a delta size.\n\nJust to be clear, my final \"it looks like a noop\" means \"good, it looks\nlike it is a pure cosmetic change and no change to the behavior.\"\n\n-Peff\n"},{"id":"343521","messageId":"20180331101750.GB32290@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8DFpqa5DYmNpBpeWOW1a3UH64AHa2vG3nkJVLoMR73wqQ@mail.gmail.com","subject":"Re: [PATCH v7 08/13] pack-objects: shrink z_delta_size field in struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-31T10:17:51Z","receivedAt":"2018-03-31T10:17:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 31, 2018 at 06:40:23AM +0200, Duy Nguyen wrote:\n\n> > Unlike the depth, I don't think there's any _inherent_ reason you\n> > couldn't throw, say, 1MB deltas into the cache (if you sized it large\n> > enough). But I doubt such deltas are really all that common. Here are\n> > the top 10 in linux.git:\n> >\n> > $ git cat-file --batch-all-objects --batch-check='%(deltabase) %(objectsize:disk)' |\n> > grep -v ^00000 | sort -k 2nr | head\n> > a02b6794337286bc12c907c33d5d75537c240bd0 769103\n> > b28d4b64c05da02c5e8c684dcb9422876225ebdc 327116\n> > 1e98ce86ed19aff9ba721d13a749ff08088c9922 325257\n> > a02b6794337286bc12c907c33d5d75537c240bd0 240647\n> > c550d99286c01867dfb26e432417f3106acf8611 177896\n> > 5977795854f852c2b95dd023fd03cace023ee41c 119737\n> > 4ccf9681c45d01d17376f7e0d266532a4460f5f8 112671\n> > b39fb6821faa9e7bc36de738152a2817b4bf3654 112657\n> > 2645d6239b74bebd661436762e819b831095b084 103980\n> > b8ce7fe5d8def58dc63b7ae099eff7bd07e4e845 101014\n> >\n> > It's possible some weird workload would want to tweak this. Say you were\n> > storing a ton of delta-capable files that were big and always differed\n> > by a megabyte. And it was somehow really important to you to tradeoff\n> > memory for CPU during the write phase of a pack.\n> \n> We're not short on spare bits so I will try to raise this limit to 1MB\n> (not because you mentioned 1MB, but because the largest size in your\n> output is close to 1MB).\n\nI doubt it matters much. Unless somebody has been tweaking the config\nthemselves, this has been limited to 1000 for everybody running\nlinux.git and nobody has ever noticed.\n\nSo I think it would only be an issue if:\n\n 1. you had an oddball repo with gigantic deltas\n\nAND\n\n 2. you for some reason really cared about caching the deltas between\n phases\n\nAND\n\n 3. you had done enough homework to even figure out that this knob\n existed\n\nI was thinking that you might care about (2) for serving fetches of your\noddball repository. But really, if you care about minimizing work, you\nwant to be reusing on-disk deltas anyway, which would skip this cache\nentirely. So any work we do to reproduce the delta would probably be\ndwarfed by the finding of this giant delta in the first place.\n\nSo raise the limit if you want, but I'd be surprised if anybody was even\ndoing (3) in the first place.\n\n-Peff\n"},{"id":"343522","messageId":"20180331102007.GC32290@sigill.intra.peff.net","threadId":"47943","inReplyTo":"CACsJy8Dma-xrDcK1vQ-s4F+Dwt4oWOQ89yr33OPJdO1g0ryQWw@mail.gmail.com","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-31T10:20:07Z","receivedAt":"2018-03-31T10:20:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 31, 2018 at 06:51:10AM +0200, Duy Nguyen wrote:\n\n> >> +#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n> >\n> > How come this one gets a macro, but the earlier conversions don't?\n> >\n> > I guess the problem is that oe_in_pack() is defined in the generic\n> > pack-objects.h, but &to_pack is only in builtin/pack-objects.c?\n> >\n> > I wonder if it would be that bad to just say oe_in_pack(&to_pack, obj)\n> > everywhere. It's longer, but it makes the code slightly less magical to\n> > read.\n> \n> Longer was exactly why I added these macros (with the hope that the\n> macro upper case names already ring a \"it's magical\" bell). Should I\n> drop all these macros? Some code becomes a lot more verbose though.\n\nI'm on the fence. I agree that the macro screams \"magical\". I just\nsometimes see a macro and think something really weird and\nunfunction-like is going on. But really we're just replacing a default\nparameter.\n\nSo I dunno. If you get rid of the macros and I look at it, I give even\nodds that I'll say \"yech, put them back!\". :)\n\n-Peff\n"},{"id":"343523","messageId":"20180331104544.GA16426@duynguyen.home","threadId":"47943","inReplyTo":"20180331102007.GC32290@sigill.intra.peff.net","subject":"Re: [PATCH v7 06/13] pack-objects: move in_pack out of struct object_entry","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T10:45:44Z","receivedAt":"2018-03-31T10:45:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 31, 2018 at 06:20:07AM -0400, Jeff King wrote:\n> On Sat, Mar 31, 2018 at 06:51:10AM +0200, Duy Nguyen wrote:\n> \n> > >> +#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n> > >\n> > > How come this one gets a macro, but the earlier conversions don't?\n> > >\n> > > I guess the problem is that oe_in_pack() is defined in the generic\n> > > pack-objects.h, but &to_pack is only in builtin/pack-objects.c?\n> > >\n> > > I wonder if it would be that bad to just say oe_in_pack(&to_pack, obj)\n> > > everywhere. It's longer, but it makes the code slightly less magical to\n> > > read.\n> > \n> > Longer was exactly why I added these macros (with the hope that the\n> > macro upper case names already ring a \"it's magical\" bell). Should I\n> > drop all these macros? Some code becomes a lot more verbose though.\n> \n> I'm on the fence. I agree that the macro screams \"magical\". I just\n> sometimes see a macro and think something really weird and\n> unfunction-like is going on. But really we're just replacing a default\n> parameter.\n> \n> So I dunno. If you get rid of the macros and I look at it, I give even\n> odds that I'll say \"yech, put them back!\". :)\n\nIt would look like this (on top of v8). I think the \"&to_pack\" part is\nmost distracting when it's used as part of an expression (or a\nfunction argument). I probably went overboard with SET_ macros though.\n\n-- 8< --\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b5bba2c228..dec849b755 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -29,18 +29,6 @@\n #include \"list.h\"\n #include \"packfile.h\"\n \n-#define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n-#define SIZE(obj) oe_size(&to_pack, obj)\n-#define SET_SIZE(obj,size) oe_set_size(&to_pack, obj, size)\n-#define DELTA_SIZE(obj) oe_delta_size(&to_pack, obj)\n-#define DELTA(obj) oe_delta(&to_pack, obj)\n-#define DELTA_CHILD(obj) oe_delta_child(&to_pack, obj)\n-#define DELTA_SIBLING(obj) oe_delta_sibling(&to_pack, obj)\n-#define SET_DELTA(obj, val) oe_set_delta(&to_pack, obj, val)\n-#define SET_DELTA_SIZE(obj, val) oe_set_delta_size(&to_pack, obj, val)\n-#define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val)\n-#define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n-\n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n \tN_(\"git pack-objects [...] [< | < ]\"),\n@@ -137,14 +125,14 @@ static void *get_delta(struct object_entry *entry)\n \tbuf = read_sha1_file(entry->idx.oid.hash, &type, &size);\n \tif (!buf)\n \t\tdie(\"unable to read %s\", oid_to_hex(&entry->idx.oid));\n-\tbase_buf = read_sha1_file(DELTA(entry)->idx.oid.hash, &type,\n+\tbase_buf = read_sha1_file(oe_delta(&to_pack, entry)->idx.oid.hash, &type,\n \t\t\t\t &base_size);\n \tif (!base_buf)\n \t\tdie(\"unable to read %s\",\n-\t\t oid_to_hex(&DELTA(entry)->idx.oid));\n+\t\t oid_to_hex(&oe_delta(&to_pack, entry)->idx.oid));\n \tdelta_buf = diff_delta(base_buf, base_size,\n \t\t\t buf, size, &delta_size, 0);\n-\tif (!delta_buf || delta_size != DELTA_SIZE(entry))\n+\tif (!delta_buf || delta_size != oe_delta_size(&to_pack, entry))\n \t\tdie(\"delta size changed\");\n \tfree(buf);\n \tfree(base_buf);\n@@ -295,15 +283,15 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tFREE_AND_NULL(entry->delta_data);\n \t\tentry->z_delta_size = 0;\n \t} else if (entry->delta_data) {\n-\t\tsize = DELTA_SIZE(entry);\n+\t\tsize = oe_delta_size(&to_pack, entry);\n \t\tbuf = entry->delta_data;\n \t\tentry->delta_data = NULL;\n-\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n+\t\ttype = (allow_ofs_delta && oe_delta(&to_pack, entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t} else {\n \t\tbuf = get_delta(entry);\n-\t\tsize = DELTA_SIZE(entry);\n-\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n+\t\tsize = oe_delta_size(&to_pack, entry);\n+\t\ttype = (allow_ofs_delta && oe_delta(&to_pack, entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n@@ -327,7 +315,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t * encoding of the relative offset for the delta\n \t\t * base from this object's position in the pack.\n \t\t */\n-\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - oe_delta(&to_pack, entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -353,7 +341,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n+\t\thashwrite(f, oe_delta(&to_pack, entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -379,7 +367,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\tunsigned long limit, int usable_delta)\n {\n-\tstruct packed_git *p = IN_PACK(entry);\n+\tstruct packed_git *p = oe_in_pack(&to_pack, entry);\n \tstruct pack_window *w_curs = NULL;\n \tstruct revindex_entry *revidx;\n \toff_t offset;\n@@ -388,10 +376,10 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n-\tunsigned long entry_size = SIZE(entry);\n+\tunsigned long entry_size = oe_size(&to_pack, entry);\n \n-\tif (DELTA(entry))\n-\t\ttype = (allow_ofs_delta && DELTA(entry)->idx.offset) ?\n+\tif (oe_delta(&to_pack, entry))\n+\t\ttype = (allow_ofs_delta && oe_delta(&to_pack, entry)->idx.offset) ?\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \thdrlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t type, entry_size);\n@@ -419,7 +407,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t}\n \n \tif (type == OBJ_OFS_DELTA) {\n-\t\toff_t ofs = entry->idx.offset - DELTA(entry)->idx.offset;\n+\t\toff_t ofs = entry->idx.offset - oe_delta(&to_pack, entry)->idx.offset;\n \t\tunsigned pos = sizeof(dheader) - 1;\n \t\tdheader[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n@@ -438,7 +426,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\treturn 0;\n \t\t}\n \t\thashwrite(f, header, hdrlen);\n-\t\thashwrite(f, DELTA(entry)->idx.oid.hash, 20);\n+\t\thashwrite(f, oe_delta(&to_pack, entry)->idx.oid.hash, 20);\n \t\thdrlen += 20;\n \t\treused_delta++;\n \t} else {\n@@ -478,20 +466,20 @@ static off_t write_object(struct hashfile *f,\n \telse\n \t\tlimit = pack_size_limit - write_offset;\n \n-\tif (!DELTA(entry))\n+\tif (!oe_delta(&to_pack, entry))\n \t\tusable_delta = 0;\t/* no delta */\n \telse if (!pack_size_limit)\n \t usable_delta = 1;\t/* unlimited packfile */\n-\telse if (DELTA(entry)->idx.offset == (off_t)-1)\n+\telse if (oe_delta(&to_pack, entry)->idx.offset == (off_t)-1)\n \t\tusable_delta = 0;\t/* base was written to another pack */\n-\telse if (DELTA(entry)->idx.offset)\n+\telse if (oe_delta(&to_pack, entry)->idx.offset)\n \t\tusable_delta = 1;\t/* base already exists in this pack */\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n \n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n-\telse if (!IN_PACK(entry))\n+\telse if (!oe_in_pack(&to_pack, entry))\n \t\tto_reuse = 0;\t/* can't reuse what we don't have */\n \telse if (oe_type(entry) == OBJ_REF_DELTA ||\n \t\t oe_type(entry) == OBJ_OFS_DELTA)\n@@ -500,7 +488,7 @@ static off_t write_object(struct hashfile *f,\n \t\t\t\t/* ... but pack split may override that */\n \telse if (oe_type(entry) != entry->in_pack_type)\n \t\tto_reuse = 0;\t/* pack has delta which is unusable */\n-\telse if (DELTA(entry))\n+\telse if (oe_delta(&to_pack, entry))\n \t\tto_reuse = 0;\t/* we want to pack afresh */\n \telse\n \t\tto_reuse = 1;\t/* we have it in-pack undeltified,\n@@ -552,12 +540,12 @@ static enum write_one_status write_one(struct hashfile *f,\n \t}\n \n \t/* if we are deltified, write out base object first. */\n-\tif (DELTA(e)) {\n+\tif (oe_delta(&to_pack, e)) {\n \t\te->idx.offset = 1; /* now recurse */\n-\t\tswitch (write_one(f, DELTA(e), offset)) {\n+\t\tswitch (write_one(f, oe_delta(&to_pack, e), offset)) {\n \t\tcase WRITE_ONE_RECURSIVE:\n \t\t\t/* we cannot depend on this one */\n-\t\t\tSET_DELTA(e, NULL);\n+\t\t\toe_set_delta(&to_pack, e, NULL);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tbreak;\n@@ -619,34 +607,34 @@ static void add_descendants_to_write_order(struct object_entry **wo,\n \t\t\t/* add this node... */\n \t\t\tadd_to_write_order(wo, endp, e);\n \t\t\t/* all its siblings... */\n-\t\t\tfor (s = DELTA_SIBLING(e); s; s = DELTA_SIBLING(s)) {\n+\t\t\tfor (s = oe_delta_sibling(&to_pack, e); s; s = oe_delta_sibling(&to_pack, s)) {\n \t\t\t\tadd_to_write_order(wo, endp, s);\n \t\t\t}\n \t\t}\n \t\t/* drop down a level to add left subtree nodes if possible */\n-\t\tif (DELTA_CHILD(e)) {\n+\t\tif (oe_delta_child(&to_pack, e)) {\n \t\t\tadd_to_order = 1;\n-\t\t\te = DELTA_CHILD(e);\n+\t\t\te = oe_delta_child(&to_pack, e);\n \t\t} else {\n \t\t\tadd_to_order = 0;\n \t\t\t/* our sibling might have some children, it is next */\n-\t\t\tif (DELTA_SIBLING(e)) {\n-\t\t\t\te = DELTA_SIBLING(e);\n+\t\t\tif (oe_delta_sibling(&to_pack, e)) {\n+\t\t\t\te = oe_delta_sibling(&to_pack, e);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\t/* go back to our parent node */\n-\t\t\te = DELTA(e);\n-\t\t\twhile (e && !DELTA_SIBLING(e)) {\n+\t\t\te = oe_delta(&to_pack, e);\n+\t\t\twhile (e && !oe_delta_sibling(&to_pack, e)) {\n \t\t\t\t/* we're on the right side of a subtree, keep\n \t\t\t\t * going up until we can go right again */\n-\t\t\t\te = DELTA(e);\n+\t\t\t\te = oe_delta(&to_pack, e);\n \t\t\t}\n \t\t\tif (!e) {\n \t\t\t\t/* done- we hit our original root node */\n \t\t\t\treturn;\n \t\t\t}\n \t\t\t/* pass it off to sibling at this level */\n-\t\t\te = DELTA_SIBLING(e);\n+\t\t\te = oe_delta_sibling(&to_pack, e);\n \t\t}\n \t};\n }\n@@ -657,7 +645,7 @@ static void add_family_to_write_order(struct object_entry **wo,\n {\n \tstruct object_entry *root;\n \n-\tfor (root = e; DELTA(root); root = DELTA(root))\n+\tfor (root = e; oe_delta(&to_pack, root); root = oe_delta(&to_pack, root))\n \t\t; /* nothing */\n \tadd_descendants_to_write_order(wo, endp, root);\n }\n@@ -672,8 +660,8 @@ static struct object_entry **compute_write_order(void)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tobjects[i].tagged = 0;\n \t\tobjects[i].filled = 0;\n-\t\tSET_DELTA_CHILD(&objects[i], NULL);\n-\t\tSET_DELTA_SIBLING(&objects[i], NULL);\n+\t\toe_set_delta_child(&to_pack, &objects[i], NULL);\n+\t\toe_set_delta_size(&to_pack, &objects[i], NULL);\n \t}\n \n \t/*\n@@ -683,11 +671,11 @@ static struct object_entry **compute_write_order(void)\n \t */\n \tfor (i = to_pack.nr_objects; i > 0;) {\n \t\tstruct object_entry *e = &objects[--i];\n-\t\tif (!DELTA(e))\n+\t\tif (!oe_delta(&to_pack, e))\n \t\t\tcontinue;\n \t\t/* Mark me as the first child */\n-\t\te->delta_sibling_idx = DELTA(e)->delta_child_idx;\n-\t\tSET_DELTA_CHILD(DELTA(e), e);\n+\t\te->delta_sibling_idx = oe_delta(&to_pack, e)->delta_child_idx;\n+\t\toe_set_delta_child(&to_pack, oe_delta(&to_pack, e), e);\n \t}\n \n \t/*\n@@ -1414,8 +1402,8 @@ static void check_object(struct object_entry *entry)\n {\n \tunsigned long canonical_size;\n \n-\tif (IN_PACK(entry)) {\n-\t\tstruct packed_git *p = IN_PACK(entry);\n+\tif (oe_in_pack(&to_pack, entry)) {\n+\t\tstruct packed_git *p = oe_in_pack(&to_pack, entry);\n \t\tstruct pack_window *w_curs = NULL;\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n@@ -1451,7 +1439,7 @@ static void check_object(struct object_entry *entry)\n \t\tdefault:\n \t\t\t/* Not a delta hence we've already got all we need. */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tSET_SIZE(entry, in_pack_size);\n+\t\t\toe_set_size(&to_pack, entry, in_pack_size);\n \t\t\tentry->in_pack_header_size = used;\n \t\t\tif (oe_type(entry) < OBJ_COMMIT || oe_type(entry) > OBJ_BLOB)\n \t\t\t\tgoto give_up;\n@@ -1508,11 +1496,11 @@ static void check_object(struct object_entry *entry)\n \t\t\t * circular deltas.\n \t\t\t */\n \t\t\toe_set_type(entry, entry->in_pack_type);\n-\t\t\tSET_SIZE(entry, in_pack_size); /* delta size */\n-\t\t\tSET_DELTA(entry, base_entry);\n-\t\t\tSET_DELTA_SIZE(entry, in_pack_size);\n+\t\t\toe_set_size(&to_pack, entry, in_pack_size); /* delta size */\n+\t\t\toe_set_delta(&to_pack, entry, base_entry);\n+\t\t\toe_set_delta_size(&to_pack, entry, in_pack_size);\n \t\t\tentry->delta_sibling_idx = base_entry->delta_child_idx;\n-\t\t\tSET_DELTA_CHILD(base_entry, entry);\n+\t\t\toe_set_delta_child(&to_pack, base_entry, entry);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1529,7 +1517,7 @@ static void check_object(struct object_entry *entry)\n \t\t\tcanonical_size = get_size_from_delta(p, &w_curs, delta_pos);\n \t\t\tif (canonical_size == 0)\n \t\t\t\tgoto give_up;\n-\t\t\tSET_SIZE(entry, canonical_size);\n+\t\t\toe_set_size(&to_pack, entry, canonical_size);\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\t}\n@@ -1546,7 +1534,7 @@ static void check_object(struct object_entry *entry)\n \toe_set_type(entry, sha1_object_info(entry->idx.oid.hash,\n \t\t\t\t\t &canonical_size));\n \tif (entry->type_valid) {\n-\t\tSET_SIZE(entry, canonical_size);\n+\t\toe_set_size(&to_pack, entry, canonical_size);\n \t} else {\n \t\t/*\n \t\t * Bad object type is checked in prepare_pack(). This is\n@@ -1561,8 +1549,8 @@ static int pack_offset_sort(const void *_a, const void *_b)\n {\n \tconst struct object_entry *a = *(struct object_entry **)_a;\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n-\tconst struct packed_git *a_in_pack = IN_PACK(a);\n-\tconst struct packed_git *b_in_pack = IN_PACK(b);\n+\tconst struct packed_git *a_in_pack = oe_in_pack(&to_pack, a);\n+\tconst struct packed_git *b_in_pack = oe_in_pack(&to_pack, b);\n \n \t/* avoid filesystem trashing with loose objects */\n \tif (!a_in_pack && !b_in_pack)\n@@ -1604,12 +1592,12 @@ static void drop_reused_delta(struct object_entry *entry)\n \t\telse\n \t\t\tidx = &oe->delta_sibling_idx;\n \t}\n-\tSET_DELTA(entry, NULL);\n+\toe_set_delta(&to_pack, entry, NULL);\n \tentry->depth = 0;\n \n \toi.sizep = &size;\n \toi.typep = &type;\n-\tif (packed_object_info(IN_PACK(entry), entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(oe_in_pack(&to_pack, entry), entry->in_pack_offset, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\n@@ -1621,7 +1609,7 @@ static void drop_reused_delta(struct object_entry *entry)\n \t} else {\n \t\toe_set_type(entry, type);\n \t}\n-\tSET_SIZE(entry, size);\n+\toe_set_size(&to_pack, entry, size);\n }\n \n /*\n@@ -1645,7 +1633,7 @@ static void break_delta_chains(struct object_entry *entry)\n \n \tfor (cur = entry, total_depth = 0;\n \t cur;\n-\t cur = DELTA(cur), total_depth++) {\n+\t cur = oe_delta(&to_pack, cur), total_depth++) {\n \t\tif (cur->dfs_state == DFS_DONE) {\n \t\t\t/*\n \t\t\t * We've already seen this object and know it isn't\n@@ -1670,7 +1658,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * it's not a delta, we're done traversing, but we'll mark it\n \t\t * done to save time on future traversals.\n \t\t */\n-\t\tif (!DELTA(cur)) {\n+\t\tif (!oe_delta(&to_pack, cur)) {\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n \t\t}\n@@ -1693,7 +1681,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t\t * We keep all commits in the chain that we examined.\n \t\t */\n \t\tcur->dfs_state = DFS_ACTIVE;\n-\t\tif (DELTA(cur)->dfs_state == DFS_ACTIVE) {\n+\t\tif (oe_delta(&to_pack, cur)->dfs_state == DFS_ACTIVE) {\n \t\t\tdrop_reused_delta(cur);\n \t\t\tcur->dfs_state = DFS_DONE;\n \t\t\tbreak;\n@@ -1708,7 +1696,7 @@ static void break_delta_chains(struct object_entry *entry)\n \t * an extra \"next\" pointer to keep going after we reset cur->delta.\n \t */\n \tfor (cur = entry; cur; cur = next) {\n-\t\tnext = DELTA(cur);\n+\t\tnext = oe_delta(&to_pack, cur);\n \n \t\t/*\n \t\t * We should have a chain of zero or more ACTIVE states down to\n@@ -1791,8 +1779,8 @@ static int type_size_sort(const void *_a, const void *_b)\n \tconst struct object_entry *b = *(struct object_entry **)_b;\n \tenum object_type a_type = oe_type(a);\n \tenum object_type b_type = oe_type(b);\n-\tunsigned long a_size = SIZE(a);\n-\tunsigned long b_size = SIZE(b);\n+\tunsigned long a_size = oe_size(&to_pack, a);\n+\tunsigned long b_size = oe_size(&to_pack, b);\n \n \tif (a_type > b_type)\n \t\treturn -1;\n@@ -1923,8 +1911,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t * it, we will still save the transfer cost, as we already know\n \t * the other side has it and we won't send src_entry at all.\n \t */\n-\tif (reuse_delta && IN_PACK(trg_entry) &&\n-\t IN_PACK(trg_entry) == IN_PACK(src_entry) &&\n+\tif (reuse_delta && oe_in_pack(&to_pack, trg_entry) &&\n+\t oe_in_pack(&to_pack, trg_entry) == oe_in_pack(&to_pack, src_entry) &&\n \t !src_entry->preferred_base &&\n \t trg_entry->in_pack_type != OBJ_REF_DELTA &&\n \t trg_entry->in_pack_type != OBJ_OFS_DELTA)\n@@ -1935,19 +1923,19 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n-\ttrg_size = SIZE(trg_entry);\n-\tif (!DELTA(trg_entry)) {\n+\ttrg_size = oe_size(&to_pack, trg_entry);\n+\tif (!oe_delta(&to_pack, trg_entry)) {\n \t\tmax_size = trg_size/2 - 20;\n \t\tref_depth = 1;\n \t} else {\n-\t\tmax_size = DELTA_SIZE(trg_entry);\n+\t\tmax_size = oe_delta_size(&to_pack, trg_entry);\n \t\tref_depth = trg->depth;\n \t}\n \tmax_size = (uint64_t)max_size * (max_depth - src->depth) /\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n-\tsrc_size = SIZE(src_entry);\n+\tsrc_size = oe_size(&to_pack, src_entry);\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -2016,9 +2004,9 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \t}\n \n-\tif (DELTA(trg_entry)) {\n+\tif (oe_delta(&to_pack, trg_entry)) {\n \t\t/* Prefer only shallower same-sized deltas. */\n-\t\tif (delta_size == DELTA_SIZE(trg_entry) &&\n+\t\tif (delta_size == oe_delta_size(&to_pack, trg_entry) &&\n \t\t src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n@@ -2033,7 +2021,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tfree(trg_entry->delta_data);\n \tcache_lock();\n \tif (trg_entry->delta_data) {\n-\t\tdelta_cache_size -= DELTA_SIZE(trg_entry);\n+\t\tdelta_cache_size -= oe_delta_size(&to_pack, trg_entry);\n \t\ttrg_entry->delta_data = NULL;\n \t}\n \tif (delta_cacheable(src_size, trg_size, delta_size)) {\n@@ -2045,8 +2033,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tfree(delta_buf);\n \t}\n \n-\tSET_DELTA(trg_entry, src_entry);\n-\tSET_DELTA_SIZE(trg_entry, delta_size);\n+\toe_set_delta(&to_pack, trg_entry, src_entry);\n+\toe_set_delta_size(&to_pack, trg_entry, delta_size);\n \ttrg->depth = src->depth + 1;\n \n \treturn 1;\n@@ -2054,13 +2042,13 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n {\n-\tstruct object_entry *child = DELTA_CHILD(me);\n+\tstruct object_entry *child = oe_delta_child(&to_pack, me);\n \tunsigned int m = n;\n \twhile (child) {\n \t\tunsigned int c = check_delta_limit(child, n + 1);\n \t\tif (m < c)\n \t\t\tm = c;\n-\t\tchild = DELTA_SIBLING(child);\n+\t\tchild = oe_delta_sibling(&to_pack, child);\n \t}\n \treturn m;\n }\n@@ -2071,7 +2059,7 @@ static unsigned long free_unpacked(struct unpacked *n)\n \tfree_delta_index(n->index);\n \tn->index = NULL;\n \tif (n->data) {\n-\t\tfreed_mem += SIZE(n->entry);\n+\t\tfreed_mem += oe_size(&to_pack, n->entry);\n \t\tFREE_AND_NULL(n->data);\n \t}\n \tn->entry = NULL;\n@@ -2129,7 +2117,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * otherwise they would become too deep.\n \t\t */\n \t\tmax_depth = depth;\n-\t\tif (DELTA_CHILD(entry)) {\n+\t\tif (oe_delta_child(&to_pack, entry)) {\n \t\t\tmax_depth -= check_delta_limit(entry, 0);\n \t\t\tif (max_depth <= 0)\n \t\t\t\tgoto next;\n@@ -2169,11 +2157,11 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\tif (entry->delta_data && !pack_to_stdout) {\n \t\t\tunsigned long size;\n \n-\t\t\tsize = do_compress(&entry->delta_data, DELTA_SIZE(entry));\n+\t\t\tsize = do_compress(&entry->delta_data, oe_delta_size(&to_pack, entry));\n \t\t\tif (size < (1U << OE_Z_DELTA_BITS)) {\n \t\t\t\tentry->z_delta_size = size;\n \t\t\t\tcache_lock();\n-\t\t\t\tdelta_cache_size -= DELTA_SIZE(entry);\n+\t\t\t\tdelta_cache_size -= oe_delta_size(&to_pack, entry);\n \t\t\t\tdelta_cache_size += entry->z_delta_size;\n \t\t\t\tcache_unlock();\n \t\t\t} else {\n@@ -2186,7 +2174,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * depth, leaving it in the window is pointless. we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (DELTA(entry) && max_depth <= n->depth)\n+\t\tif (oe_delta(&to_pack, entry) && max_depth <= n->depth)\n \t\t\tcontinue;\n \n \t\t/*\n@@ -2194,7 +2182,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n \t\t * currently deltified object, to keep it longer. It will\n \t\t * be the first base object to be attempted next.\n \t\t */\n-\t\tif (DELTA(entry)) {\n+\t\tif (oe_delta(&to_pack, entry)) {\n \t\t\tstruct unpacked swap = array[best_base];\n \t\t\tint dist = (window + idx - best_base) % window;\n \t\t\tint dst = best_base;\n@@ -2515,7 +2503,7 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (DELTA(entry))\n+\t\tif (oe_delta(&to_pack, entry))\n \t\t\t/* This happens if we decided to reuse existing\n \t\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n \t\t\t */\n-- 8< --\n"},{"id":"343524","messageId":"87efk0wk7f.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"Re: [PATCH v8 00/15] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-31T11:36:20Z","receivedAt":"2018-03-31T11:36:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 31 2018, Nguyễn Thái Ngọc Duy wrote:\n\nI'm testing this and it looks good to me so far, aside from this:\n\n> - use git_env_*() instead of manually handling getenv() values\n> [...]\n> \tstruct packed_git **mapping, *p;\n> -\tint cnt = 0, nr = 1 << OE_IN_PACK_BITS;\n> -\n> -\tif (getenv(\"GIT_TEST_FULL_IN_PACK_ARRAY\")) {\n> -\t\t/*\n> -\t\t * leave in_pack_by_idx NULL to force in_pack[] to be\n> -\t\t * used instead\n> -\t\t */\n> -\t\treturn;\n> -\t}\n> [...]\n>\n> +\tif (git_env_bool(\"GIT_TEST_FULL_IN_PACK_ARRAY\", 0)) {\n> +\t\t/*\n> +\t\t * do not initialize in_pack_by_idx[] to force the\n> +\t\t * slow path in oe_in_pack()\n> +\t\t */\n> +\t} else {\n> +\t\tprepare_in_pack_by_idx(pdata);\n> +\t}\n> [...]\n> diff --git a/t/README b/t/README\n> index 02bfb3fed5..c01d210c15 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -291,16 +291,26 @@ expect the rest to function correctly.\n> and know what setup is needed for it. Or when you want to run\n> everything up to a certain test.\n>\n> +\n> +Running tests with special setups\n> +---------------------------------\n> +\n> +The whole test suite could be run to test some special features\n> +that cannot be easily covered by a few specific test cases. These\n> +could be enabled by running the test suite with correct GIT_TEST_\n> +environment set.\n> +\n> +GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n> +\n> GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n> path where there are more than 1024 packs even if the actual number of\n> packs in repository is below this limit.\n>\n> -GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n> -code path where we do not cache objecct size in memory and read it\n> -from existing packs on demand. This normally only happens when the\n> -object size is over 2GB. This variable forces the code path on any\n> -object larger than 2^ bytes.\n\nThe docs here say set these env variables, but actually\nGIT_TEST_FULL_IN_PACK_ARRAY is a special snowflake in requiring you to\nset a bool value.\n\nI'd set GIT_TEST_SPLIT_INDEX=YesPlease already in my test setup & just\ncopied that as GIT_TEST_FULL_IN_PACK_ARRAY=YesPlease, but that'll error\nout since it's expecting bool, not the env variable to be set.\n\nI really don't care which we use, but let's use either if(getenv()) or\nif(git_env_bool()) consistently, and then have the docs either say \"if\nset\" or \"if set to a boolean value (see git-config(1))\".\n"},{"id":"343525","messageId":"CACsJy8AAApXdxPwt6Yoc-gESXFLVNW8O2u_LF6HUBJTAQaCsTw@mail.gmail.com","threadId":"47943","inReplyTo":"87efk0wk7f.fsf@evledraar.gmail.com","subject":"Re: [PATCH v8 00/15] nd/pack-objects-pack-struct updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-03-31T12:08:09Z","receivedAt":"2018-03-31T12:08:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 31, 2018 at 1:36 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>> +GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n>> +\n>> GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n>> path where there are more than 1024 packs even if the actual number of\n>> packs in repository is below this limit.\n>>\n>> -GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n>> -code path where we do not cache objecct size in memory and read it\n>> -from existing packs on demand. This normally only happens when the\n>> -object size is over 2GB. This variable forces the code path on any\n>> -object larger than 2^ bytes.\n>\n> The docs here say set these env variables, but actually\n> GIT_TEST_FULL_IN_PACK_ARRAY is a special snowflake in requiring you to\n> set a bool value.\n>\n> I'd set GIT_TEST_SPLIT_INDEX=YesPlease already in my test setup & just\n> copied that as GIT_TEST_FULL_IN_PACK_ARRAY=YesPlease, but that'll error\n> out since it's expecting bool, not the env variable to be set.\n>\n> I really don't care which we use, but let's use either if(getenv()) or\n> if(git_env_bool()) consistently, and then have the docs either say \"if\n> set\" or \"if set to a boolean value (see git-config(1))\".\n\nI'll change GIT_TEST_SPLIT_INDEX to boolean too since I document it\nhere anyway. Will wait for a while though to see if anything else\nshould be part of v9.\n-- \nDuy\n"},{"id":"343535","messageId":"87d0zkw8r9.fsf@evledraar.gmail.com","threadId":"47943","inReplyTo":"CACsJy8AAApXdxPwt6Yoc-gESXFLVNW8O2u_LF6HUBJTAQaCsTw@mail.gmail.com","subject":"Re: [PATCH v8 00/15] nd/pack-objects-pack-struct updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-31T15:43:38Z","receivedAt":"2018-03-31T15:43:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 31 2018, Duy Nguyen wrote:\n\n> On Sat, Mar 31, 2018 at 1:36 PM, Ævar Arnfjörð Bjarmason\n> wrote:\n>>> +GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.\n>>> +\n>>> GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code\n>>> path where there are more than 1024 packs even if the actual number of\n>>> packs in repository is below this limit.\n>>>\n>>> -GIT_TEST_OE_SIZE_BITS= exercises the uncommon pack-objects\n>>> -code path where we do not cache objecct size in memory and read it\n>>> -from existing packs on demand. This normally only happens when the\n>>> -object size is over 2GB. This variable forces the code path on any\n>>> -object larger than 2^ bytes.\n>>\n>> The docs here say set these env variables, but actually\n>> GIT_TEST_FULL_IN_PACK_ARRAY is a special snowflake in requiring you to\n>> set a bool value.\n>>\n>> I'd set GIT_TEST_SPLIT_INDEX=YesPlease already in my test setup & just\n>> copied that as GIT_TEST_FULL_IN_PACK_ARRAY=YesPlease, but that'll error\n>> out since it's expecting bool, not the env variable to be set.\n>>\n>> I really don't care which we use, but let's use either if(getenv()) or\n>> if(git_env_bool()) consistently, and then have the docs either say \"if\n>> set\" or \"if set to a boolean value (see git-config(1))\".\n>\n> I'll change GIT_TEST_SPLIT_INDEX to boolean too since I document it\n> here anyway. Will wait for a while though to see if anything else\n> should be part of v9.\n\nSounds good, FWIW (since I spied your forced push to your private branch\non Github) I mean something like this on top of what you just pushed:\n\n diff --git a/t/README b/t/README\n index 65dee935c0..583bede192 100644\n --- a/t/README\n +++ b/t/README\n @@ -298,7 +298,8 @@ Running tests with special setups\n The whole test suite could be run to test some special features\n that cannot be easily covered by a few specific test cases. These\n could be enabled by running the test suite with correct GIT_TEST_\n -environment set.\n +environment variable set to a boolean value, as documented in the\n +\"Values\" section of git-config(1).\n\n GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole\n test suite.\n\nI.e. the part above where we just say it has to be set should be changed\nto indicate it's a boolean as understood by git, since in shell/*nix\nidiom saying something has to be set just means ensure getenv() won't\nreturn NULL.\n"},{"id":"344016","messageId":"20180406214739.GG7870@sigill.intra.peff.net","threadId":"47943","inReplyTo":"20180331100311.32373-1-pclouds@gmail.com","subject":"Re: [PATCH v8 00/15] nd/pack-objects-pack-struct updates","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-06T21:47:39Z","receivedAt":"2018-04-06T21:47:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 31, 2018 at 12:02:56PM +0200, Nguyễn Thái Ngọc Duy wrote:\n\n> v8 changes\n> \n> - prefer BUG() over die()\n> - do \"1U <<\" instead of \"1 << \" to avoid undefined behavior with\n> signed shifting.\n> - add more comments based on Jeff's feedback\n> - plug a leak in try_delta() when delta_size is too large\n> - be kind and set depth/cache_max_small_delta_size to max limit\n> instead of dying when the user gives a value over limit\n> - make travis execute pack-objects uncommon code\n> - use git_env_*() instead of manually handling getenv() values\n> - fallback code for when a new pack is added when pack-objects is\n> running\n> - Compressed cached delta size limit is increased from 64k to 1MB\n> - Cached delta size limit is decreased from 2G to 1MB\n\nI ran out of time to give this a very careful review, and I'm trying to\nclear my plate before going offline for a few weeks. I did look at the\ninterdiff and it seemed sane.\n\nSo I think going forward, you can at least consider my objections\nretracted. I don't promise this iteration is bug-free, but I think\nyou've addressed all my earlier issues. :)\n\nThanks.\n\n-Peff\n"}]}