{"thread":{"id":"38310","subject":"low memory system to clone larger repo","startedAt":"2015-01-08T16:10:08Z","lastAt":"2015-02-11T13:10:05Z","messageCount":12,"participants":["matthew sporleder","Duy Nguyen","Matt Sporleder","Nguyễn Thái Ngọc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254466","messageId":"CAHKF-AspyE84_0CVMz2OjFLt3Q62qKDfTkbUk3-+RQ_EZ=0JGg@mail.gmail.com","threadId":"38310","inReplyTo":null,"subject":"low memory system to clone larger repo","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2015-01-08T16:10:08Z","receivedAt":"2015-01-08T16:10:08Z","isPatch":false,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"I am attempting to clone this repo: https://github.com/jsonn/src/\n\nand have been successful on some lower memory systems, but i'm\ninterested in continuing to push down the limit.\n\nI am getting more success running clone via https:// than git:// or\nssh (which is confusing to me) and the smallest system that works is a\nraspberry pi with 256 RAM + 256 swap.\n\nI seem to run out of memory consistently around 16% into Resolving\ndeltas phase but I don't notice an RSS jump so that's another\nconfusing spot.\n\nMy config is below and I'd appreciate any more suggestions of getting\nthat down to working on a 128MB box (or smaller).\n\n---\n\nI appreciate any suggestions,\nMatt\n\np.s. shallow clones work fine on very small systems\n\n\n[pack]\n        windowMemory = 1m\n        packSizeLimit = 1m\n        deltaCacheSize = 1m\n        deltaCacheLimit = 10\n        packSizeLimit = 1m\n        threads = 1\n[core]\n        packedGitWindowSize = 1m\n        packedGitLimit = 1m\n        deltaBaseCacheLimit = 1m\n        compression = 0\n        loosecompression = 0\n        bigFileThreshold = 10m\n[http]\n        sslVerify = false\n[transfer]\n        unpackLimit = 10\n"},{"id":"255814","messageId":"CACsJy8Cx6K3Qdq4hq7T_vxsOR-UJv7+mz9AFSiAeKd3YZxqYHg@mail.gmail.com","threadId":"38310","inReplyTo":"CAHKF-AspyE84_0CVMz2OjFLt3Q62qKDfTkbUk3-+RQ_EZ=0JGg@mail.gmail.com","subject":"Re: low memory system to clone larger repo","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-09T10:40:49Z","receivedAt":"2015-02-09T10:40:49Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:\n> I am attempting to clone this repo: https://github.com/jsonn/src/\n>\n> and have been successful on some lower memory systems, but i'm\n> interested in continuing to push down the limit.\n>\n> I am getting more success running clone via https:// than git:// or\n> ssh (which is confusing to me) and the smallest system that works is a\n> raspberry pi with 256 RAM + 256 swap.\n>\n> I seem to run out of memory consistently around 16% into Resolving\n> deltas phase but I don't notice an RSS jump so that's another\n> confusing spot.\n\nSorry for a really late reply. The command that's running when you run\nout of memory is index-pack. I guess it's verifying the delta chain. I\nthink it needs enough memory for two uncompressed objects (or files)\nin a delta chain. I haven't finished cloning this repo yet so I don't\nknow what these delta chains look like.\n\nWhat does it say when it runs out of memory? I'm thinking maybe we\ncould force a core dump, then look at how memory is used.\n\nWhat if you \"git init\", then do \"git fetch https://...\" manually?\nThere's an optimization for git-clone that may make index-pack use a\nbit more memory (and push it over the edge)\n\n> My config is below and I'd appreciate any more suggestions of getting\n> that down to working on a 128MB box (or smaller).\n\nI suppose it's ~/.gitconfig or /etc/gitconfig, it's not added after\nthe clone is complete, correct? Sounds interesting, let me profile its\nmemory usage..\n\n> [pack]\n>         windowMemory = 1m\n>         packSizeLimit = 1m\n>         deltaCacheSize = 1m\n>         deltaCacheLimit = 10\n>         packSizeLimit = 1m\n\nI think many of these only affect the server side. If you clone from\ngithub, then they are useless. You may want to provide your own server\nside with these settings and see if things change. Also play with\npack.depth (affecting server side)\n\n>         threads = 1\n-- \nDuy\n"},{"id":"255815","messageId":"EF215DDC-22ED-426B-9C8D-5BA91E6EEACB@gmail.com","threadId":"38310","inReplyTo":"CACsJy8Cx6K3Qdq4hq7T_vxsOR-UJv7+mz9AFSiAeKd3YZxqYHg@mail.gmail.com","subject":"Re: low memory system to clone larger repo","fromName":"Matt Sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2015-02-09T11:20:07Z","receivedAt":"2015-02-09T11:20:07Z","isPatch":false,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"A more \"final\" version of the tuning exercise I did is here:\n\nhttp://mail-index.netbsd.org/tech-repository/2015/01/08/msg000520.html\n\nI did try some of these setting on the server and it made the repo much much larger so I guess I am looking for ways to just reduce client memory usage/the best balance of disk, memory, and bandwidth. \n\nIf there is a way to turn off some memory hogging speed ups I m interested in trying them. \n\nI will try to get you the output later since I have since started working on the server side.\n\nLet me know if you want to try some server side stuff and I can give you a git:// or http:// off list. \n\nThanks for looking, \nMatt\n\n\n> On Feb 9, 2015, at 5:40 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> \n>> On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:\n>> I am attempting to clone this repo: https://github.com/jsonn/src/\n>> \n>> and have been successful on some lower memory systems, but i'm\n>> interested in continuing to push down the limit.\n>> \n>> I am getting more success running clone via https:// than git:// or\n>> ssh (which is confusing to me) and the smallest system that works is a\n>> raspberry pi with 256 RAM + 256 swap.\n>> \n>> I seem to run out of memory consistently around 16% into Resolving\n>> deltas phase but I don't notice an RSS jump so that's another\n>> confusing spot.\n> \n> Sorry for a really late reply. The command that's running when you run\n> out of memory is index-pack. I guess it's verifying the delta chain. I\n> think it needs enough memory for two uncompressed objects (or files)\n> in a delta chain. I haven't finished cloning this repo yet so I don't\n> know what these delta chains look like.\n> \n> What does it say when it runs out of memory? I'm thinking maybe we\n> could force a core dump, then look at how memory is used.\n> \n> What if you \"git init\", then do \"git fetch https://...\" manually?\n> There's an optimization for git-clone that may make index-pack use a\n> bit more memory (and push it over the edge)\n> \n>> My config is below and I'd appreciate any more suggestions of getting\n>> that down to working on a 128MB box (or smaller).\n> \n> I suppose it's ~/.gitconfig or /etc/gitconfig, it's not added after\n> the clone is complete, correct? Sounds interesting, let me profile its\n> memory usage..\n> \n>> [pack]\n>>        windowMemory = 1m\n>>        packSizeLimit = 1m\n>>        deltaCacheSize = 1m\n>>        deltaCacheLimit = 10\n>>        packSizeLimit = 1m\n> \n> I think many of these only affect the server side. If you clone from\n> github, then they are useless. You may want to provide your own server\n> side with these settings and see if things change. Also play with\n> pack.depth (affecting server side)\n> \n>>        threads = 1\n> -- \n> Duy\n"},{"id":"255817","messageId":"CACsJy8A=6m5sWnDhPPMNrWbZ=fOMXPxO_1GVh-WpHycf5gm+rg@mail.gmail.com","threadId":"38310","inReplyTo":"CAHKF-AspyE84_0CVMz2OjFLt3Q62qKDfTkbUk3-+RQ_EZ=0JGg@mail.gmail.com","subject":"Re: low memory system to clone larger repo","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-09T12:32:19Z","receivedAt":"2015-02-09T12:32:19Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:\n> I am attempting to clone this repo: https://github.com/jsonn/src/\n\nThis repo has 3.4M objects. Basic book keeping would cost 200MB (in\npractice it'll be higher because I'm assuming no deltas in my\ncalculation). On my 64-bit system, it already uses 400+ MB at the\nbeginning of delta resolving phase, and is about 500MB during. 32-bit\nsystems cost less but I doubt we could keep it within 256 MB limit. I\nthink you just need more powerful machines for a repo this size.\n\nAlso, they have some large files (udivmodti4_test.c 16MB, MD5SUMS\n6MB..) These giant files could make index-pack use more memory\nespecially if they are deltified. If you repack the repo with\ncore.bigFileThreshold about 1-2MB, then clone, you may get a better\nmemory consumption, but at the cost of bigger packs.\n-- \nDuy\n"},{"id":"255819","messageId":"1423487929-28019-1-git-send-email-pclouds@gmail.com","threadId":"38310","inReplyTo":"CACsJy8A=6m5sWnDhPPMNrWbZ=fOMXPxO_1GVh-WpHycf5gm+rg@mail.gmail.com","subject":"[PATCH] index-pack: reduce memory footprint a bit","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-09T13:18:49Z","receivedAt":"2015-02-09T13:18:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"For each object in the input pack, we need one struct object_entry. On\nx86-64, this struct is 64 bytes long. Although:\n\n - The 8 bytes for delta_depth and base_object_no are only useful when\n   show_stat is set. And it's never set unless someone is debugging.\n\n - The three fields hdr_size, type and real_type take 4 bytes each\n   even though they never use more than 4 bits.\n\nBy moving delta_depth and base_object_no out of struct object_entry\nand make the other 3 fields one byte long instead of 4, we shrink 25%\nof this struct.\n\nOn a 3.4M object repo that's about 53MB. The saving is less impressive\ncompared to index-pack total memory use (about 400MB before delta\nresolving, so the saving is just 13%)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n I'm not sure if this patch is worth pursuing. It makes the code a\n little bit harder to read. I was just wondering how much memory could\n be saved..\n\n We could maybe save some more by splitting union delta_base with the\n assumption that pack-objects would utilize delta-ofs-offset as much\n as possible, which makes the delta_base.sha1[] a waste most of the\n time.\n \n This repo has 2803447 deltas, and because it's a clone case, all\n delta should be ofs-delta, which means we waste about 32MB. But\n shrinking this could get ugly.\n\n builtin/index-pack.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4632117..479ec5e 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -18,9 +18,12 @@ static const char index_pack_usage[] =\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\n-\tunsigned int hdr_size;\n-\tenum object_type type;\n-\tenum object_type real_type;\n+\tunsigned char hdr_size;\n+\tchar type;\n+\tchar real_type;\n+};\n+\n+struct object_entry_extra {\n \tunsigned delta_depth;\n \tint base_object_no;\n };\n@@ -64,6 +67,7 @@ struct delta_entry {\n };\n \n static struct object_entry *objects;\n+static struct object_entry_extra *objects_extra;\n static struct delta_entry *deltas;\n static struct thread_local nothread_data;\n static int nr_objects;\n@@ -873,13 +877,15 @@ static void resolve_delta(struct object_entry *delta_obj,\n \tvoid *base_data, *delta_data;\n \n \tif (show_stat) {\n-\t\tdelta_obj->delta_depth = base->obj->delta_depth + 1;\n+\t\tint i = delta_obj - objects;\n+\t\tint j = base->obj - objects;\n+\t\tobjects_extra[i].delta_depth = objects_extra[j].delta_depth + 1;\n \t\tdeepest_delta_lock();\n-\t\tif (deepest_delta < delta_obj->delta_depth)\n-\t\t\tdeepest_delta = delta_obj->delta_depth;\n+\t\tif (deepest_delta < objects_extra[i].delta_depth)\n+\t\t\tdeepest_delta = objects_extra[i].delta_depth;\n \t\tdeepest_delta_unlock();\n+\t\tobjects_extra[i].base_object_no = j;\n \t}\n-\tdelta_obj->base_object_no = base->obj - objects;\n \tdelta_data = get_data_from_pack(delta_obj);\n \tbase_data = get_base_data(base);\n \tresult->obj = delta_obj;\n@@ -902,7 +908,7 @@ static void resolve_delta(struct object_entry *delta_obj,\n  * \"want\"; if so, swap in \"set\" and return true. Otherwise, leave it untouched\n  * and return false.\n  */\n-static int compare_and_swap_type(enum object_type *type,\n+static int compare_and_swap_type(char *type,\n \t\t\t\t enum object_type want,\n \t\t\t\t enum object_type set)\n {\n@@ -1499,7 +1505,7 @@ static void show_pack_info(int stat_only)\n \t\tstruct object_entry *obj = &objects[i];\n \n \t\tif (is_delta_type(obj->type))\n-\t\t\tchain_histogram[obj->delta_depth - 1]++;\n+\t\t\tchain_histogram[objects_extra[i].delta_depth - 1]++;\n \t\tif (stat_only)\n \t\t\tcontinue;\n \t\tprintf(\"%s %-6s %lu %lu %\"PRIuMAX,\n@@ -1508,8 +1514,8 @@ static void show_pack_info(int stat_only)\n \t\t       (unsigned long)(obj[1].idx.offset - obj->idx.offset),\n \t\t       (uintmax_t)obj->idx.offset);\n \t\tif (is_delta_type(obj->type)) {\n-\t\t\tstruct object_entry *bobj = &objects[obj->base_object_no];\n-\t\t\tprintf(\" %u %s\", obj->delta_depth, sha1_to_hex(bobj->idx.sha1));\n+\t\t\tstruct object_entry *bobj = &objects[objects_extra[i].base_object_no];\n+\t\t\tprintf(\" %u %s\", objects_extra[i].delta_depth, sha1_to_hex(bobj->idx.sha1));\n \t\t}\n \t\tputchar('\\n');\n \t}\n@@ -1672,6 +1678,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tcurr_pack = open_pack_file(pack_name);\n \tparse_pack_header();\n \tobjects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n+\tif (show_stat)\n+\t\tobjects_extra = xcalloc(nr_objects + 1, sizeof(struct object_entry_extra));\n \tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n \tparse_pack_objects(pack_sha1);\n \tresolve_deltas();\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"255840","messageId":"xmqqfvaec2cm.fsf@gitster.dls.corp.google.com","threadId":"38310","inReplyTo":"1423487929-28019-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-09T19:27:21Z","receivedAt":"2015-02-09T19:27:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:\n\n> For each object in the input pack, we need one struct object_entry. On\n> x86-64, this struct is 64 bytes long. Although:\n>\n>  - The 8 bytes for delta_depth and base_object_no are only useful when\n>    show_stat is set. And it's never set unless someone is debugging.\n>\n>  - The three fields hdr_size, type and real_type take 4 bytes each\n>    even though they never use more than 4 bits.\n>\n> By moving delta_depth and base_object_no out of struct object_entry\n> and make the other 3 fields one byte long instead of 4, we shrink 25%\n> of this struct.\n>\n> On a 3.4M object repo that's about 53MB. The saving is less impressive\n> compared to index-pack total memory use (about 400MB before delta\n> resolving, so the saving is just 13%)\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n>  I'm not sure if this patch is worth pursuing. It makes the code a\n>  little bit harder to read. I was just wondering how much memory could\n>  be saved..\n\nI would say 13% is already impressive ;-).\n\nI do not find the result all that harder to read.  I however think\nthat the change would make it a lot harder to maintain, especially\nbecause the name \"object-entry-extra\" does not have any direct link\nto \"show-stat\" to hint us that this must be allocated when show-stat\nis in use and must never be looked at when show-stat is not in use.\n\nAlso it makes me wonder if the compilers are smart enough to notice\nthat the codepaths that access objects_extra[] are OK because they\nare all inside \"if (show_stat)\".\n"},{"id":"255876","messageId":"CAHKF-At1ybThZ54yNJZQDWooMcLPeCi-QKAi4AjzZyiK86_dOA@mail.gmail.com","threadId":"38310","inReplyTo":"CACsJy8A=6m5sWnDhPPMNrWbZ=fOMXPxO_1GVh-WpHycf5gm+rg@mail.gmail.com","subject":"Re: low memory system to clone larger repo","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2015-02-10T03:56:48Z","receivedAt":"2015-02-10T03:56:48Z","isPatch":false,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"Below is the output from my index-pack/clone over git://\n\nThis is with the recent memory patch applied but testing some less crazy tuning.\n--\n\nThe corruption of the signal number shows up in google from other\npeople so I guess it's a lingering bug.\n\n*\ngit-tests $ git clone git://github.com/jsonn/src\nCloning into 'src'...\nremote: Counting objects: 3497569, done.\nremote: Compressing objects: 100% (640647/640647), done.\nerror: index-pack died of signal 9497569), 990.62 MiB | 8.35 MiB/s\nfatal: index-pack failed\n\n\n\n\n\n\nOn Mon, Feb 9, 2015 at 6:32 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:\n>> I am attempting to clone this repo: https://github.com/jsonn/src/\n>\n> This repo has 3.4M objects. Basic book keeping would cost 200MB (in\n> practice it'll be higher because I'm assuming no deltas in my\n> calculation). On my 64-bit system, it already uses 400+ MB at the\n> beginning of delta resolving phase, and is about 500MB during. 32-bit\n> systems cost less but I doubt we could keep it within 256 MB limit. I\n> think you just need more powerful machines for a repo this size.\n>\n> Also, they have some large files (udivmodti4_test.c 16MB, MD5SUMS\n> 6MB..) These giant files could make index-pack use more memory\n> especially if they are deltified. If you repack the repo with\n> core.bigFileThreshold about 1-2MB, then clone, you may get a better\n> memory consumption, but at the cost of bigger packs.\n> --\n> Duy\n"},{"id":"255878","messageId":"20150210093041.GA30992@lanh","threadId":"38310","inReplyTo":"xmqqfvaec2cm.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-10T09:30:41Z","receivedAt":"2015-02-10T09:30:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Feb 09, 2015 at 11:27:21AM -0800, Junio C Hamano wrote:\n> > On a 3.4M object repo that's about 53MB. The saving is less impressive\n> > compared to index-pack total memory use (about 400MB before delta\n> > resolving, so the saving is just 13%)\n> >\n> > Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> > ---\n> >  I'm not sure if this patch is worth pursuing. It makes the code a\n> >  little bit harder to read. I was just wondering how much memory could\n> >  be saved..\n\n(text reordered)\n\n> I do not find the result all that harder to read.  I however think\n> that the change would make it a lot harder to maintain, especially\n> because the name \"object-entry-extra\" does not have any direct link\n> to \"show-stat\" to hint us that this must be allocated when show-stat\n> is in use and must never be looked at when show-stat is not in use.\n\nNoted. To be fixed.\n\n> I would say 13% is already impressive ;-).\n\nThe second patch makes the total saving 119MB, close to 30% (again on\nx86-64, 32-bit platform number may be different). If we only compare\nwith the size of objects[] and deltas[], the saving percentage is 37%\n(only for clone case) for this repo. Now it looks impressive to me :-D\n\nThe patch is larger than the previous one, but not really complex. And\nthe final index-pack.c is not hard to read either, probably becase we\nalready handle ofs-delta and ref-delta separately.\n\n-- 8< --\nSubject: [PATCH 2/2] index-pack: kill union delta_base to save memory\n\nOnce we know the number of objects in the input pack, we allocate an\narray of nr_objects of struct delta_entry. On x86-64, this struct is\n32 bytes long. The union delta_base, which is part of struct\ndelta_entry, provides enough space to store either ofs-delta (8 bytes)\nor ref-delta (20 bytes).\n\nNotice that with \"recent\" Git versions, ofs-delta objects are\npreferred over ref-delta objects and ref-delta objects have no reason\nto be present in a clone pack. So in clone case we waste\n(20-8) * nr_objects bytes because of this union. That's about 38MB out\nof 100MB for deltas[] with 3.4M objects, or 38%. deltas[] would be\naround 62MB without the waste.\n\nThis patch attempts to eliminate that. deltas[] array is split into\ntwo: one for ofs-delta and one for ref-delta. Many functions are also\nduplicated because of this split. With this patch, ofs_delta_entry[]\narray takes 38MB. ref_deltas[] should remain unallocated in clone case\n(0 bytes). This array grows as we see ref-delta. We save more than\nhalf in clone case, or 25% of total book keeping.\n\nThe saving is more than the calculation above because padding is\nremoved by __attribute__((packed)) on ofs_delta_entry. This attribute\nshould be ok to use, as we used to have it in our code base for some\ntime. The last use was removed because it may lead to incorrect\nbehavior when the struct is not packed, which is not the case in\nindex-pack.\n\nA note about ofs_deltas allocation. We could use ref_deltas memory\nallocation strategy for ofs_deltas. But that probably just adds more\noverhead on top. ofs-deltas are generally the majority (1/2 to 2/3) in\nany pack. Incremental realloc may lead to too many memcpy. And if we\npreallocate, say 1/2 or 2/3 of nr_objects initially, the growth rate\nof ALLOC_GROW() could make this array larger than nr_objects, wasting\nmore memory.\n\nBrought-up-by: Matthew Sporleder <msporleder@gmail.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/index-pack.c | 260 +++++++++++++++++++++++++++++++--------------------\n 1 file changed, 160 insertions(+), 100 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 07b2c0c..27e3c8b 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -28,11 +28,6 @@ struct object_stat {\n \tint base_object_no;\n };\n \n-union delta_base {\n-\tunsigned char sha1[20];\n-\toff_t offset;\n-};\n-\n struct base_data {\n \tstruct base_data *base;\n \tstruct base_data *child;\n@@ -52,26 +47,28 @@ struct thread_local {\n \tint pack_fd;\n };\n \n-/*\n- * Even if sizeof(union delta_base) == 24 on 64-bit archs, we really want\n- * to memcmp() only the first 20 bytes.\n- */\n-#define UNION_BASE_SZ\t20\n-\n #define FLAG_LINK (1u<<20)\n #define FLAG_CHECKED (1u<<21)\n \n-struct delta_entry {\n-\tunion delta_base base;\n+struct ofs_delta_entry {\n+\toff_t offset;\n+\tint obj_no;\n+} __attribute__((packed));\n+\n+struct ref_delta_entry {\n+\tunsigned char sha1[20];\n \tint obj_no;\n };\n \n static struct object_entry *objects;\n static struct object_stat *obj_stat;\n-static struct delta_entry *deltas;\n+static struct ofs_delta_entry *ofs_deltas;\n+static struct ref_delta_entry *ref_deltas;\n static struct thread_local nothread_data;\n static int nr_objects;\n-static int nr_deltas;\n+static int nr_ofs_deltas;\n+static int nr_ref_deltas;\n+static int ref_deltas_alloc;\n static int nr_resolved_deltas;\n static int nr_threads;\n \n@@ -480,7 +477,8 @@ static void *unpack_entry_data(unsigned long offset, unsigned long size,\n }\n \n static void *unpack_raw_entry(struct object_entry *obj,\n-\t\t\t      union delta_base *delta_base,\n+\t\t\t      off_t *ofs_offset,\n+\t\t\t      unsigned char *ref_sha1,\n \t\t\t      unsigned char *sha1)\n {\n \tunsigned char *p;\n@@ -509,11 +507,10 @@ static void *unpack_raw_entry(struct object_entry *obj,\n \n \tswitch (obj->type) {\n \tcase OBJ_REF_DELTA:\n-\t\thashcpy(delta_base->sha1, fill(20));\n+\t\thashcpy(ref_sha1, fill(20));\n \t\tuse(20);\n \t\tbreak;\n \tcase OBJ_OFS_DELTA:\n-\t\tmemset(delta_base, 0, sizeof(*delta_base));\n \t\tp = fill(1);\n \t\tc = *p;\n \t\tuse(1);\n@@ -527,8 +524,8 @@ static void *unpack_raw_entry(struct object_entry *obj,\n \t\t\tuse(1);\n \t\t\tbase_offset = (base_offset << 7) + (c & 127);\n \t\t}\n-\t\tdelta_base->offset = obj->idx.offset - base_offset;\n-\t\tif (delta_base->offset <= 0 || delta_base->offset >= obj->idx.offset)\n+\t\t*ofs_offset = obj->idx.offset - base_offset;\n+\t\tif (*ofs_offset <= 0 || *ofs_offset >= obj->idx.offset)\n \t\t\tbad_object(obj->idx.offset, _(\"delta base offset is out of bound\"));\n \t\tbreak;\n \tcase OBJ_COMMIT:\n@@ -612,55 +609,108 @@ static void *get_data_from_pack(struct object_entry *obj)\n \treturn unpack_data(obj, NULL, NULL);\n }\n \n-static int compare_delta_bases(const union delta_base *base1,\n-\t\t\t       const union delta_base *base2,\n-\t\t\t       enum object_type type1,\n-\t\t\t       enum object_type type2)\n+static int compare_ofs_delta_bases(off_t offset1, off_t offset2,\n+\t\t\t\t   enum object_type type1,\n+\t\t\t\t   enum object_type type2)\n+{\n+\tint cmp = type1 - type2;\n+\tif (cmp)\n+\t\treturn cmp;\n+\treturn offset1 - offset2;\n+}\n+\n+static int find_ofs_delta(const off_t offset, enum object_type type)\n+{\n+\tint first = 0, last = nr_ofs_deltas;\n+\n+\twhile (first < last) {\n+\t\tint next = (first + last) / 2;\n+\t\tstruct ofs_delta_entry *delta = &ofs_deltas[next];\n+\t\tint cmp;\n+\n+\t\tcmp = compare_ofs_delta_bases(offset, delta->offset,\n+\t\t\t\t\t      type, objects[delta->obj_no].type);\n+\t\tif (!cmp)\n+\t\t\treturn next;\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\treturn -first-1;\n+}\n+\n+static void find_ofs_delta_children(off_t offset,\n+\t\t\t\t    int *first_index, int *last_index,\n+\t\t\t\t    enum object_type type)\n+{\n+\tint first = find_ofs_delta(offset, type);\n+\tint last = first;\n+\tint end = nr_ofs_deltas - 1;\n+\n+\tif (first < 0) {\n+\t\t*first_index = 0;\n+\t\t*last_index = -1;\n+\t\treturn;\n+\t}\n+\twhile (first > 0 && ofs_deltas[first - 1].offset == offset)\n+\t\t--first;\n+\twhile (last < end && ofs_deltas[last + 1].offset == offset)\n+\t\t++last;\n+\t*first_index = first;\n+\t*last_index = last;\n+}\n+\n+static int compare_ref_delta_bases(const unsigned char *sha1,\n+\t\t\t\t   const unsigned char *sha2,\n+\t\t\t\t   enum object_type type1,\n+\t\t\t\t   enum object_type type2)\n {\n \tint cmp = type1 - type2;\n \tif (cmp)\n \t\treturn cmp;\n-\treturn memcmp(base1, base2, UNION_BASE_SZ);\n+\treturn hashcmp(sha1, sha2);\n }\n \n-static int find_delta(const union delta_base *base, enum object_type type)\n+static int find_ref_delta(const unsigned char *sha1, enum object_type type)\n {\n-\tint first = 0, last = nr_deltas;\n-\n-        while (first < last) {\n-                int next = (first + last) / 2;\n-                struct delta_entry *delta = &deltas[next];\n-                int cmp;\n-\n-\t\tcmp = compare_delta_bases(base, &delta->base,\n-\t\t\t\t\t  type, objects[delta->obj_no].type);\n-                if (!cmp)\n-                        return next;\n-                if (cmp < 0) {\n-                        last = next;\n-                        continue;\n-                }\n-                first = next+1;\n-        }\n-        return -first-1;\n+\tint first = 0, last = nr_ref_deltas;\n+\n+\twhile (first < last) {\n+\t\tint next = (first + last) / 2;\n+\t\tstruct ref_delta_entry *delta = &ref_deltas[next];\n+\t\tint cmp;\n+\n+\t\tcmp = compare_ref_delta_bases(sha1, delta->sha1,\n+\t\t\t\t\t      type, objects[delta->obj_no].type);\n+\t\tif (!cmp)\n+\t\t\treturn next;\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\treturn -first-1;\n }\n \n-static void find_delta_children(const union delta_base *base,\n-\t\t\t\tint *first_index, int *last_index,\n-\t\t\t\tenum object_type type)\n+static void find_ref_delta_children(const unsigned char *sha1,\n+\t\t\t\t    int *first_index, int *last_index,\n+\t\t\t\t    enum object_type type)\n {\n-\tint first = find_delta(base, type);\n+\tint first = find_ref_delta(sha1, type);\n \tint last = first;\n-\tint end = nr_deltas - 1;\n+\tint end = nr_ref_deltas - 1;\n \n \tif (first < 0) {\n \t\t*first_index = 0;\n \t\t*last_index = -1;\n \t\treturn;\n \t}\n-\twhile (first > 0 && !memcmp(&deltas[first - 1].base, base, UNION_BASE_SZ))\n+\twhile (first > 0 && !hashcmp(ref_deltas[first - 1].sha1, sha1))\n \t\t--first;\n-\twhile (last < end && !memcmp(&deltas[last + 1].base, base, UNION_BASE_SZ))\n+\twhile (last < end && !hashcmp(ref_deltas[last + 1].sha1, sha1))\n \t\t++last;\n \t*first_index = first;\n \t*last_index = last;\n@@ -927,16 +977,13 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n \t\t\t\t\t\t  struct base_data *prev_base)\n {\n \tif (base->ref_last == -1 && base->ofs_last == -1) {\n-\t\tunion delta_base base_spec;\n-\n-\t\thashcpy(base_spec.sha1, base->obj->idx.sha1);\n-\t\tfind_delta_children(&base_spec,\n-\t\t\t\t    &base->ref_first, &base->ref_last, OBJ_REF_DELTA);\n+\t\tfind_ref_delta_children(base->obj->idx.sha1,\n+\t\t\t\t\t&base->ref_first, &base->ref_last,\n+\t\t\t\t\tOBJ_REF_DELTA);\n \n-\t\tmemset(&base_spec, 0, sizeof(base_spec));\n-\t\tbase_spec.offset = base->obj->idx.offset;\n-\t\tfind_delta_children(&base_spec,\n-\t\t\t\t    &base->ofs_first, &base->ofs_last, OBJ_OFS_DELTA);\n+\t\tfind_ofs_delta_children(base->obj->idx.offset,\n+\t\t\t\t\t&base->ofs_first, &base->ofs_last,\n+\t\t\t\t\tOBJ_OFS_DELTA);\n \n \t\tif (base->ref_last == -1 && base->ofs_last == -1) {\n \t\t\tfree(base->data);\n@@ -947,7 +994,7 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n \t}\n \n \tif (base->ref_first <= base->ref_last) {\n-\t\tstruct object_entry *child = objects + deltas[base->ref_first].obj_no;\n+\t\tstruct object_entry *child = objects + ref_deltas[base->ref_first].obj_no;\n \t\tstruct base_data *result = alloc_base_data();\n \n \t\tif (!compare_and_swap_type(&child->real_type, OBJ_REF_DELTA,\n@@ -963,7 +1010,7 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n \t}\n \n \tif (base->ofs_first <= base->ofs_last) {\n-\t\tstruct object_entry *child = objects + deltas[base->ofs_first].obj_no;\n+\t\tstruct object_entry *child = objects + ofs_deltas[base->ofs_first].obj_no;\n \t\tstruct base_data *result = alloc_base_data();\n \n \t\tassert(child->real_type == OBJ_OFS_DELTA);\n@@ -999,15 +1046,20 @@ static void find_unresolved_deltas(struct base_data *base)\n \t}\n }\n \n-static int compare_delta_entry(const void *a, const void *b)\n+static int compare_ofs_delta_entry(const void *a, const void *b)\n+{\n+\tconst struct ofs_delta_entry *delta_a = a;\n+\tconst struct ofs_delta_entry *delta_b = b;\n+\n+\treturn delta_a->offset - delta_b->offset;\n+}\n+\n+static int compare_ref_delta_entry(const void *a, const void *b)\n {\n-\tconst struct delta_entry *delta_a = a;\n-\tconst struct delta_entry *delta_b = b;\n+\tconst struct ref_delta_entry *delta_a = a;\n+\tconst struct ref_delta_entry *delta_b = b;\n \n-\t/* group by type (ref vs ofs) and then by value (sha-1 or offset) */\n-\treturn compare_delta_bases(&delta_a->base, &delta_b->base,\n-\t\t\t\t   objects[delta_a->obj_no].type,\n-\t\t\t\t   objects[delta_b->obj_no].type);\n+\treturn hashcmp(delta_a->sha1, delta_b->sha1);\n }\n \n static void resolve_base(struct object_entry *obj)\n@@ -1053,7 +1105,8 @@ static void *threaded_second_pass(void *data)\n static void parse_pack_objects(unsigned char *sha1)\n {\n \tint i, nr_delays = 0;\n-\tstruct delta_entry *delta = deltas;\n+\tstruct ofs_delta_entry *ofs_delta = ofs_deltas;\n+\tunsigned char ref_delta_sha1[20];\n \tstruct stat st;\n \n \tif (verbose)\n@@ -1062,12 +1115,18 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t\t\tnr_objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = &objects[i];\n-\t\tvoid *data = unpack_raw_entry(obj, &delta->base, obj->idx.sha1);\n+\t\tvoid *data = unpack_raw_entry(obj, &ofs_delta->offset,\n+\t\t\t\t\t      ref_delta_sha1, obj->idx.sha1);\n \t\tobj->real_type = obj->type;\n-\t\tif (is_delta_type(obj->type)) {\n-\t\t\tnr_deltas++;\n-\t\t\tdelta->obj_no = i;\n-\t\t\tdelta++;\n+\t\tif (obj->type == OBJ_OFS_DELTA) {\n+\t\t\tnr_ofs_deltas++;\n+\t\t\tofs_delta->obj_no = i;\n+\t\t\tofs_delta++;\n+\t\t} else if (obj->type == OBJ_REF_DELTA) {\n+\t\t\tALLOC_GROW(ref_deltas, nr_ref_deltas + 1, ref_deltas_alloc);\n+\t\t\thashcpy(ref_deltas[nr_ref_deltas].sha1, ref_delta_sha1);\n+\t\t\tref_deltas[nr_ref_deltas].obj_no = i;\n+\t\t\tnr_ref_deltas++;\n \t\t} else if (!data) {\n \t\t\t/* large blobs, check later */\n \t\t\tobj->real_type = OBJ_BAD;\n@@ -1118,15 +1177,18 @@ static void resolve_deltas(void)\n {\n \tint i;\n \n-\tif (!nr_deltas)\n+\tif (!nr_ofs_deltas && !nr_ref_deltas)\n \t\treturn;\n \n \t/* Sort deltas by base SHA1/offset for fast searching */\n-\tqsort(deltas, nr_deltas, sizeof(struct delta_entry),\n-\t      compare_delta_entry);\n+\tqsort(ofs_deltas, nr_ofs_deltas, sizeof(struct ofs_delta_entry),\n+\t      compare_ofs_delta_entry);\n+\tqsort(ref_deltas, nr_ref_deltas, sizeof(struct ref_delta_entry),\n+\t      compare_ref_delta_entry);\n \n \tif (verbose)\n-\t\tprogress = start_progress(_(\"Resolving deltas\"), nr_deltas);\n+\t\tprogress = start_progress(_(\"Resolving deltas\"),\n+\t\t\t\t\t  nr_ref_deltas + nr_ofs_deltas);\n \n #ifndef NO_PTHREADS\n \tnr_dispatched = 0;\n@@ -1164,7 +1226,7 @@ static void resolve_deltas(void)\n static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved);\n static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned char *pack_sha1)\n {\n-\tif (nr_deltas == nr_resolved_deltas) {\n+\tif (nr_ref_deltas + nr_ofs_deltas == nr_resolved_deltas) {\n \t\tstop_progress(&progress);\n \t\t/* Flush remaining pack final 20-byte SHA1. */\n \t\tflush();\n@@ -1175,7 +1237,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha\n \t\tstruct sha1file *f;\n \t\tunsigned char read_sha1[20], tail_sha1[20];\n \t\tstruct strbuf msg = STRBUF_INIT;\n-\t\tint nr_unresolved = nr_deltas - nr_resolved_deltas;\n+\t\tint nr_unresolved = nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas;\n \t\tint nr_objects_initial = nr_objects;\n \t\tif (nr_unresolved <= 0)\n \t\t\tdie(_(\"confusion beyond insanity\"));\n@@ -1197,11 +1259,11 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha\n \t\t\tdie(_(\"Unexpected tail checksum for %s \"\n \t\t\t      \"(disk corruption?)\"), curr_pack);\n \t}\n-\tif (nr_deltas != nr_resolved_deltas)\n+\tif (nr_ofs_deltas + nr_ref_deltas != nr_resolved_deltas)\n \t\tdie(Q_(\"pack has %d unresolved delta\",\n \t\t       \"pack has %d unresolved deltas\",\n-\t\t       nr_deltas - nr_resolved_deltas),\n-\t\t    nr_deltas - nr_resolved_deltas);\n+\t\t       nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas),\n+\t\t    nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas);\n }\n \n static int write_compressed(struct sha1file *f, void *in, unsigned int size)\n@@ -1261,14 +1323,14 @@ static struct object_entry *append_obj_to_pack(struct sha1file *f,\n \n static int delta_pos_compare(const void *_a, const void *_b)\n {\n-\tstruct delta_entry *a = *(struct delta_entry **)_a;\n-\tstruct delta_entry *b = *(struct delta_entry **)_b;\n+\tstruct ref_delta_entry *a = *(struct ref_delta_entry **)_a;\n+\tstruct ref_delta_entry *b = *(struct ref_delta_entry **)_b;\n \treturn a->obj_no - b->obj_no;\n }\n \n static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved)\n {\n-\tstruct delta_entry **sorted_by_pos;\n+\tstruct ref_delta_entry **sorted_by_pos;\n \tint i, n = 0;\n \n \t/*\n@@ -1282,28 +1344,25 @@ static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved)\n \t * resolving deltas in the same order as their position in the pack.\n \t */\n \tsorted_by_pos = xmalloc(nr_unresolved * sizeof(*sorted_by_pos));\n-\tfor (i = 0; i < nr_deltas; i++) {\n-\t\tif (objects[deltas[i].obj_no].real_type != OBJ_REF_DELTA)\n-\t\t\tcontinue;\n-\t\tsorted_by_pos[n++] = &deltas[i];\n-\t}\n+\tfor (i = 0; i < nr_ref_deltas; i++)\n+\t\tsorted_by_pos[n++] = &ref_deltas[i];\n \tqsort(sorted_by_pos, n, sizeof(*sorted_by_pos), delta_pos_compare);\n \n \tfor (i = 0; i < n; i++) {\n-\t\tstruct delta_entry *d = sorted_by_pos[i];\n+\t\tstruct ref_delta_entry *d = sorted_by_pos[i];\n \t\tenum object_type type;\n \t\tstruct base_data *base_obj = alloc_base_data();\n \n \t\tif (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n \t\t\tcontinue;\n-\t\tbase_obj->data = read_sha1_file(d->base.sha1, &type, &base_obj->size);\n+\t\tbase_obj->data = read_sha1_file(d->sha1, &type, &base_obj->size);\n \t\tif (!base_obj->data)\n \t\t\tcontinue;\n \n-\t\tif (check_sha1_signature(d->base.sha1, base_obj->data,\n+\t\tif (check_sha1_signature(d->sha1, base_obj->data,\n \t\t\t\tbase_obj->size, typename(type)))\n-\t\t\tdie(_(\"local object %s is corrupt\"), sha1_to_hex(d->base.sha1));\n-\t\tbase_obj->obj = append_obj_to_pack(f, d->base.sha1,\n+\t\t\tdie(_(\"local object %s is corrupt\"), sha1_to_hex(d->sha1));\n+\t\tbase_obj->obj = append_obj_to_pack(f, d->sha1,\n \t\t\t\t\tbase_obj->data, base_obj->size, type);\n \t\tfind_unresolved_deltas(base_obj);\n \t\tdisplay_progress(progress, nr_resolved_deltas);\n@@ -1495,7 +1554,7 @@ static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)\n \n static void show_pack_info(int stat_only)\n {\n-\tint i, baseobjects = nr_objects - nr_deltas;\n+\tint i, baseobjects = nr_objects - nr_ref_deltas - nr_ofs_deltas;\n \tunsigned long *chain_histogram = NULL;\n \n \tif (deepest_delta)\n@@ -1680,11 +1739,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tobjects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n \tif (show_stat)\n \t\tobj_stat = xcalloc(nr_objects + 1, sizeof(struct object_stat));\n-\tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n+\tofs_deltas = xcalloc(nr_objects, sizeof(struct ofs_delta_entry));\n \tparse_pack_objects(pack_sha1);\n \tresolve_deltas();\n \tconclude_pack(fix_thin_pack, curr_pack, pack_sha1);\n-\tfree(deltas);\n+\tfree(ofs_deltas);\n+\tfree(ref_deltas);\n \tif (strict)\n \t\tforeign_nr = check_objects();\n \n-- \n2.2.0.513.g477eb31\n\n-- 8< --\n"},{"id":"255881","messageId":"CAHKF-Atr_ezupL02aW08S-6NGGLi55vHuVep1mQvOaQq0Xh=FA@mail.gmail.com","threadId":"38310","inReplyTo":"20150210093041.GA30992@lanh","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2015-02-10T12:08:53Z","receivedAt":"2015-02-10T12:08:53Z","isPatch":true,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"I'm having trouble getting this new patch to apply.  Are you working\non a branch that I can track?\n\nOn Tue, Feb 10, 2015 at 3:30 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Mon, Feb 09, 2015 at 11:27:21AM -0800, Junio C Hamano wrote:\n>> > On a 3.4M object repo that's about 53MB. The saving is less impressive\n>> > compared to index-pack total memory use (about 400MB before delta\n>> > resolving, so the saving is just 13%)\n>> >\n>> > Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n>> > ---\n>> >  I'm not sure if this patch is worth pursuing. It makes the code a\n>> >  little bit harder to read. I was just wondering how much memory could\n>> >  be saved..\n>\n> (text reordered)\n>\n>> I do not find the result all that harder to read.  I however think\n>> that the change would make it a lot harder to maintain, especially\n>> because the name \"object-entry-extra\" does not have any direct link\n>> to \"show-stat\" to hint us that this must be allocated when show-stat\n>> is in use and must never be looked at when show-stat is not in use.\n>\n> Noted. To be fixed.\n>\n>> I would say 13% is already impressive ;-).\n>\n> The second patch makes the total saving 119MB, close to 30% (again on\n> x86-64, 32-bit platform number may be different). If we only compare\n> with the size of objects[] and deltas[], the saving percentage is 37%\n> (only for clone case) for this repo. Now it looks impressive to me :-D\n>\n> The patch is larger than the previous one, but not really complex. And\n> the final index-pack.c is not hard to read either, probably becase we\n> already handle ofs-delta and ref-delta separately.\n>\n> -- 8< --\n> Subject: [PATCH 2/2] index-pack: kill union delta_base to save memory\n>\n> Once we know the number of objects in the input pack, we allocate an\n> array of nr_objects of struct delta_entry. On x86-64, this struct is\n> 32 bytes long. The union delta_base, which is part of struct\n> delta_entry, provides enough space to store either ofs-delta (8 bytes)\n> or ref-delta (20 bytes).\n>\n> Notice that with \"recent\" Git versions, ofs-delta objects are\n> preferred over ref-delta objects and ref-delta objects have no reason\n> to be present in a clone pack. So in clone case we waste\n> (20-8) * nr_objects bytes because of this union. That's about 38MB out\n> of 100MB for deltas[] with 3.4M objects, or 38%. deltas[] would be\n> around 62MB without the waste.\n>\n> This patch attempts to eliminate that. deltas[] array is split into\n> two: one for ofs-delta and one for ref-delta. Many functions are also\n> duplicated because of this split. With this patch, ofs_delta_entry[]\n> array takes 38MB. ref_deltas[] should remain unallocated in clone case\n> (0 bytes). This array grows as we see ref-delta. We save more than\n> half in clone case, or 25% of total book keeping.\n>\n> The saving is more than the calculation above because padding is\n> removed by __attribute__((packed)) on ofs_delta_entry. This attribute\n> should be ok to use, as we used to have it in our code base for some\n> time. The last use was removed because it may lead to incorrect\n> behavior when the struct is not packed, which is not the case in\n> index-pack.\n>\n> A note about ofs_deltas allocation. We could use ref_deltas memory\n> allocation strategy for ofs_deltas. But that probably just adds more\n> overhead on top. ofs-deltas are generally the majority (1/2 to 2/3) in\n> any pack. Incremental realloc may lead to too many memcpy. And if we\n> preallocate, say 1/2 or 2/3 of nr_objects initially, the growth rate\n> of ALLOC_GROW() could make this array larger than nr_objects, wasting\n> more memory.\n>\n> Brought-up-by: Matthew Sporleder <msporleder@gmail.com>\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n>  builtin/index-pack.c | 260 +++++++++++++++++++++++++++++++--------------------\n>  1 file changed, 160 insertions(+), 100 deletions(-)\n>\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 07b2c0c..27e3c8b 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -28,11 +28,6 @@ struct object_stat {\n>         int base_object_no;\n>  };\n>\n> -union delta_base {\n> -       unsigned char sha1[20];\n> -       off_t offset;\n> -};\n> -\n>  struct base_data {\n>         struct base_data *base;\n>         struct base_data *child;\n> @@ -52,26 +47,28 @@ struct thread_local {\n>         int pack_fd;\n>  };\n>\n> -/*\n> - * Even if sizeof(union delta_base) == 24 on 64-bit archs, we really want\n> - * to memcmp() only the first 20 bytes.\n> - */\n> -#define UNION_BASE_SZ  20\n> -\n>  #define FLAG_LINK (1u<<20)\n>  #define FLAG_CHECKED (1u<<21)\n>\n> -struct delta_entry {\n> -       union delta_base base;\n> +struct ofs_delta_entry {\n> +       off_t offset;\n> +       int obj_no;\n> +} __attribute__((packed));\n> +\n> +struct ref_delta_entry {\n> +       unsigned char sha1[20];\n>         int obj_no;\n>  };\n>\n>  static struct object_entry *objects;\n>  static struct object_stat *obj_stat;\n> -static struct delta_entry *deltas;\n> +static struct ofs_delta_entry *ofs_deltas;\n> +static struct ref_delta_entry *ref_deltas;\n>  static struct thread_local nothread_data;\n>  static int nr_objects;\n> -static int nr_deltas;\n> +static int nr_ofs_deltas;\n> +static int nr_ref_deltas;\n> +static int ref_deltas_alloc;\n>  static int nr_resolved_deltas;\n>  static int nr_threads;\n>\n> @@ -480,7 +477,8 @@ static void *unpack_entry_data(unsigned long offset, unsigned long size,\n>  }\n>\n>  static void *unpack_raw_entry(struct object_entry *obj,\n> -                             union delta_base *delta_base,\n> +                             off_t *ofs_offset,\n> +                             unsigned char *ref_sha1,\n>                               unsigned char *sha1)\n>  {\n>         unsigned char *p;\n> @@ -509,11 +507,10 @@ static void *unpack_raw_entry(struct object_entry *obj,\n>\n>         switch (obj->type) {\n>         case OBJ_REF_DELTA:\n> -               hashcpy(delta_base->sha1, fill(20));\n> +               hashcpy(ref_sha1, fill(20));\n>                 use(20);\n>                 break;\n>         case OBJ_OFS_DELTA:\n> -               memset(delta_base, 0, sizeof(*delta_base));\n>                 p = fill(1);\n>                 c = *p;\n>                 use(1);\n> @@ -527,8 +524,8 @@ static void *unpack_raw_entry(struct object_entry *obj,\n>                         use(1);\n>                         base_offset = (base_offset << 7) + (c & 127);\n>                 }\n> -               delta_base->offset = obj->idx.offset - base_offset;\n> -               if (delta_base->offset <= 0 || delta_base->offset >= obj->idx.offset)\n> +               *ofs_offset = obj->idx.offset - base_offset;\n> +               if (*ofs_offset <= 0 || *ofs_offset >= obj->idx.offset)\n>                         bad_object(obj->idx.offset, _(\"delta base offset is out of bound\"));\n>                 break;\n>         case OBJ_COMMIT:\n> @@ -612,55 +609,108 @@ static void *get_data_from_pack(struct object_entry *obj)\n>         return unpack_data(obj, NULL, NULL);\n>  }\n>\n> -static int compare_delta_bases(const union delta_base *base1,\n> -                              const union delta_base *base2,\n> -                              enum object_type type1,\n> -                              enum object_type type2)\n> +static int compare_ofs_delta_bases(off_t offset1, off_t offset2,\n> +                                  enum object_type type1,\n> +                                  enum object_type type2)\n> +{\n> +       int cmp = type1 - type2;\n> +       if (cmp)\n> +               return cmp;\n> +       return offset1 - offset2;\n> +}\n> +\n> +static int find_ofs_delta(const off_t offset, enum object_type type)\n> +{\n> +       int first = 0, last = nr_ofs_deltas;\n> +\n> +       while (first < last) {\n> +               int next = (first + last) / 2;\n> +               struct ofs_delta_entry *delta = &ofs_deltas[next];\n> +               int cmp;\n> +\n> +               cmp = compare_ofs_delta_bases(offset, delta->offset,\n> +                                             type, objects[delta->obj_no].type);\n> +               if (!cmp)\n> +                       return next;\n> +               if (cmp < 0) {\n> +                       last = next;\n> +                       continue;\n> +               }\n> +               first = next+1;\n> +       }\n> +       return -first-1;\n> +}\n> +\n> +static void find_ofs_delta_children(off_t offset,\n> +                                   int *first_index, int *last_index,\n> +                                   enum object_type type)\n> +{\n> +       int first = find_ofs_delta(offset, type);\n> +       int last = first;\n> +       int end = nr_ofs_deltas - 1;\n> +\n> +       if (first < 0) {\n> +               *first_index = 0;\n> +               *last_index = -1;\n> +               return;\n> +       }\n> +       while (first > 0 && ofs_deltas[first - 1].offset == offset)\n> +               --first;\n> +       while (last < end && ofs_deltas[last + 1].offset == offset)\n> +               ++last;\n> +       *first_index = first;\n> +       *last_index = last;\n> +}\n> +\n> +static int compare_ref_delta_bases(const unsigned char *sha1,\n> +                                  const unsigned char *sha2,\n> +                                  enum object_type type1,\n> +                                  enum object_type type2)\n>  {\n>         int cmp = type1 - type2;\n>         if (cmp)\n>                 return cmp;\n> -       return memcmp(base1, base2, UNION_BASE_SZ);\n> +       return hashcmp(sha1, sha2);\n>  }\n>\n> -static int find_delta(const union delta_base *base, enum object_type type)\n> +static int find_ref_delta(const unsigned char *sha1, enum object_type type)\n>  {\n> -       int first = 0, last = nr_deltas;\n> -\n> -        while (first < last) {\n> -                int next = (first + last) / 2;\n> -                struct delta_entry *delta = &deltas[next];\n> -                int cmp;\n> -\n> -               cmp = compare_delta_bases(base, &delta->base,\n> -                                         type, objects[delta->obj_no].type);\n> -                if (!cmp)\n> -                        return next;\n> -                if (cmp < 0) {\n> -                        last = next;\n> -                        continue;\n> -                }\n> -                first = next+1;\n> -        }\n> -        return -first-1;\n> +       int first = 0, last = nr_ref_deltas;\n> +\n> +       while (first < last) {\n> +               int next = (first + last) / 2;\n> +               struct ref_delta_entry *delta = &ref_deltas[next];\n> +               int cmp;\n> +\n> +               cmp = compare_ref_delta_bases(sha1, delta->sha1,\n> +                                             type, objects[delta->obj_no].type);\n> +               if (!cmp)\n> +                       return next;\n> +               if (cmp < 0) {\n> +                       last = next;\n> +                       continue;\n> +               }\n> +               first = next+1;\n> +       }\n> +       return -first-1;\n>  }\n>\n> -static void find_delta_children(const union delta_base *base,\n> -                               int *first_index, int *last_index,\n> -                               enum object_type type)\n> +static void find_ref_delta_children(const unsigned char *sha1,\n> +                                   int *first_index, int *last_index,\n> +                                   enum object_type type)\n>  {\n> -       int first = find_delta(base, type);\n> +       int first = find_ref_delta(sha1, type);\n>         int last = first;\n> -       int end = nr_deltas - 1;\n> +       int end = nr_ref_deltas - 1;\n>\n>         if (first < 0) {\n>                 *first_index = 0;\n>                 *last_index = -1;\n>                 return;\n>         }\n> -       while (first > 0 && !memcmp(&deltas[first - 1].base, base, UNION_BASE_SZ))\n> +       while (first > 0 && !hashcmp(ref_deltas[first - 1].sha1, sha1))\n>                 --first;\n> -       while (last < end && !memcmp(&deltas[last + 1].base, base, UNION_BASE_SZ))\n> +       while (last < end && !hashcmp(ref_deltas[last + 1].sha1, sha1))\n>                 ++last;\n>         *first_index = first;\n>         *last_index = last;\n> @@ -927,16 +977,13 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n>                                                   struct base_data *prev_base)\n>  {\n>         if (base->ref_last == -1 && base->ofs_last == -1) {\n> -               union delta_base base_spec;\n> -\n> -               hashcpy(base_spec.sha1, base->obj->idx.sha1);\n> -               find_delta_children(&base_spec,\n> -                                   &base->ref_first, &base->ref_last, OBJ_REF_DELTA);\n> +               find_ref_delta_children(base->obj->idx.sha1,\n> +                                       &base->ref_first, &base->ref_last,\n> +                                       OBJ_REF_DELTA);\n>\n> -               memset(&base_spec, 0, sizeof(base_spec));\n> -               base_spec.offset = base->obj->idx.offset;\n> -               find_delta_children(&base_spec,\n> -                                   &base->ofs_first, &base->ofs_last, OBJ_OFS_DELTA);\n> +               find_ofs_delta_children(base->obj->idx.offset,\n> +                                       &base->ofs_first, &base->ofs_last,\n> +                                       OBJ_OFS_DELTA);\n>\n>                 if (base->ref_last == -1 && base->ofs_last == -1) {\n>                         free(base->data);\n> @@ -947,7 +994,7 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n>         }\n>\n>         if (base->ref_first <= base->ref_last) {\n> -               struct object_entry *child = objects + deltas[base->ref_first].obj_no;\n> +               struct object_entry *child = objects + ref_deltas[base->ref_first].obj_no;\n>                 struct base_data *result = alloc_base_data();\n>\n>                 if (!compare_and_swap_type(&child->real_type, OBJ_REF_DELTA,\n> @@ -963,7 +1010,7 @@ static struct base_data *find_unresolved_deltas_1(struct base_data *base,\n>         }\n>\n>         if (base->ofs_first <= base->ofs_last) {\n> -               struct object_entry *child = objects + deltas[base->ofs_first].obj_no;\n> +               struct object_entry *child = objects + ofs_deltas[base->ofs_first].obj_no;\n>                 struct base_data *result = alloc_base_data();\n>\n>                 assert(child->real_type == OBJ_OFS_DELTA);\n> @@ -999,15 +1046,20 @@ static void find_unresolved_deltas(struct base_data *base)\n>         }\n>  }\n>\n> -static int compare_delta_entry(const void *a, const void *b)\n> +static int compare_ofs_delta_entry(const void *a, const void *b)\n> +{\n> +       const struct ofs_delta_entry *delta_a = a;\n> +       const struct ofs_delta_entry *delta_b = b;\n> +\n> +       return delta_a->offset - delta_b->offset;\n> +}\n> +\n> +static int compare_ref_delta_entry(const void *a, const void *b)\n>  {\n> -       const struct delta_entry *delta_a = a;\n> -       const struct delta_entry *delta_b = b;\n> +       const struct ref_delta_entry *delta_a = a;\n> +       const struct ref_delta_entry *delta_b = b;\n>\n> -       /* group by type (ref vs ofs) and then by value (sha-1 or offset) */\n> -       return compare_delta_bases(&delta_a->base, &delta_b->base,\n> -                                  objects[delta_a->obj_no].type,\n> -                                  objects[delta_b->obj_no].type);\n> +       return hashcmp(delta_a->sha1, delta_b->sha1);\n>  }\n>\n>  static void resolve_base(struct object_entry *obj)\n> @@ -1053,7 +1105,8 @@ static void *threaded_second_pass(void *data)\n>  static void parse_pack_objects(unsigned char *sha1)\n>  {\n>         int i, nr_delays = 0;\n> -       struct delta_entry *delta = deltas;\n> +       struct ofs_delta_entry *ofs_delta = ofs_deltas;\n> +       unsigned char ref_delta_sha1[20];\n>         struct stat st;\n>\n>         if (verbose)\n> @@ -1062,12 +1115,18 @@ static void parse_pack_objects(unsigned char *sha1)\n>                                 nr_objects);\n>         for (i = 0; i < nr_objects; i++) {\n>                 struct object_entry *obj = &objects[i];\n> -               void *data = unpack_raw_entry(obj, &delta->base, obj->idx.sha1);\n> +               void *data = unpack_raw_entry(obj, &ofs_delta->offset,\n> +                                             ref_delta_sha1, obj->idx.sha1);\n>                 obj->real_type = obj->type;\n> -               if (is_delta_type(obj->type)) {\n> -                       nr_deltas++;\n> -                       delta->obj_no = i;\n> -                       delta++;\n> +               if (obj->type == OBJ_OFS_DELTA) {\n> +                       nr_ofs_deltas++;\n> +                       ofs_delta->obj_no = i;\n> +                       ofs_delta++;\n> +               } else if (obj->type == OBJ_REF_DELTA) {\n> +                       ALLOC_GROW(ref_deltas, nr_ref_deltas + 1, ref_deltas_alloc);\n> +                       hashcpy(ref_deltas[nr_ref_deltas].sha1, ref_delta_sha1);\n> +                       ref_deltas[nr_ref_deltas].obj_no = i;\n> +                       nr_ref_deltas++;\n>                 } else if (!data) {\n>                         /* large blobs, check later */\n>                         obj->real_type = OBJ_BAD;\n> @@ -1118,15 +1177,18 @@ static void resolve_deltas(void)\n>  {\n>         int i;\n>\n> -       if (!nr_deltas)\n> +       if (!nr_ofs_deltas && !nr_ref_deltas)\n>                 return;\n>\n>         /* Sort deltas by base SHA1/offset for fast searching */\n> -       qsort(deltas, nr_deltas, sizeof(struct delta_entry),\n> -             compare_delta_entry);\n> +       qsort(ofs_deltas, nr_ofs_deltas, sizeof(struct ofs_delta_entry),\n> +             compare_ofs_delta_entry);\n> +       qsort(ref_deltas, nr_ref_deltas, sizeof(struct ref_delta_entry),\n> +             compare_ref_delta_entry);\n>\n>         if (verbose)\n> -               progress = start_progress(_(\"Resolving deltas\"), nr_deltas);\n> +               progress = start_progress(_(\"Resolving deltas\"),\n> +                                         nr_ref_deltas + nr_ofs_deltas);\n>\n>  #ifndef NO_PTHREADS\n>         nr_dispatched = 0;\n> @@ -1164,7 +1226,7 @@ static void resolve_deltas(void)\n>  static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved);\n>  static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned char *pack_sha1)\n>  {\n> -       if (nr_deltas == nr_resolved_deltas) {\n> +       if (nr_ref_deltas + nr_ofs_deltas == nr_resolved_deltas) {\n>                 stop_progress(&progress);\n>                 /* Flush remaining pack final 20-byte SHA1. */\n>                 flush();\n> @@ -1175,7 +1237,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha\n>                 struct sha1file *f;\n>                 unsigned char read_sha1[20], tail_sha1[20];\n>                 struct strbuf msg = STRBUF_INIT;\n> -               int nr_unresolved = nr_deltas - nr_resolved_deltas;\n> +               int nr_unresolved = nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas;\n>                 int nr_objects_initial = nr_objects;\n>                 if (nr_unresolved <= 0)\n>                         die(_(\"confusion beyond insanity\"));\n> @@ -1197,11 +1259,11 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha\n>                         die(_(\"Unexpected tail checksum for %s \"\n>                               \"(disk corruption?)\"), curr_pack);\n>         }\n> -       if (nr_deltas != nr_resolved_deltas)\n> +       if (nr_ofs_deltas + nr_ref_deltas != nr_resolved_deltas)\n>                 die(Q_(\"pack has %d unresolved delta\",\n>                        \"pack has %d unresolved deltas\",\n> -                      nr_deltas - nr_resolved_deltas),\n> -                   nr_deltas - nr_resolved_deltas);\n> +                      nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas),\n> +                   nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas);\n>  }\n>\n>  static int write_compressed(struct sha1file *f, void *in, unsigned int size)\n> @@ -1261,14 +1323,14 @@ static struct object_entry *append_obj_to_pack(struct sha1file *f,\n>\n>  static int delta_pos_compare(const void *_a, const void *_b)\n>  {\n> -       struct delta_entry *a = *(struct delta_entry **)_a;\n> -       struct delta_entry *b = *(struct delta_entry **)_b;\n> +       struct ref_delta_entry *a = *(struct ref_delta_entry **)_a;\n> +       struct ref_delta_entry *b = *(struct ref_delta_entry **)_b;\n>         return a->obj_no - b->obj_no;\n>  }\n>\n>  static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved)\n>  {\n> -       struct delta_entry **sorted_by_pos;\n> +       struct ref_delta_entry **sorted_by_pos;\n>         int i, n = 0;\n>\n>         /*\n> @@ -1282,28 +1344,25 @@ static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved)\n>          * resolving deltas in the same order as their position in the pack.\n>          */\n>         sorted_by_pos = xmalloc(nr_unresolved * sizeof(*sorted_by_pos));\n> -       for (i = 0; i < nr_deltas; i++) {\n> -               if (objects[deltas[i].obj_no].real_type != OBJ_REF_DELTA)\n> -                       continue;\n> -               sorted_by_pos[n++] = &deltas[i];\n> -       }\n> +       for (i = 0; i < nr_ref_deltas; i++)\n> +               sorted_by_pos[n++] = &ref_deltas[i];\n>         qsort(sorted_by_pos, n, sizeof(*sorted_by_pos), delta_pos_compare);\n>\n>         for (i = 0; i < n; i++) {\n> -               struct delta_entry *d = sorted_by_pos[i];\n> +               struct ref_delta_entry *d = sorted_by_pos[i];\n>                 enum object_type type;\n>                 struct base_data *base_obj = alloc_base_data();\n>\n>                 if (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n>                         continue;\n> -               base_obj->data = read_sha1_file(d->base.sha1, &type, &base_obj->size);\n> +               base_obj->data = read_sha1_file(d->sha1, &type, &base_obj->size);\n>                 if (!base_obj->data)\n>                         continue;\n>\n> -               if (check_sha1_signature(d->base.sha1, base_obj->data,\n> +               if (check_sha1_signature(d->sha1, base_obj->data,\n>                                 base_obj->size, typename(type)))\n> -                       die(_(\"local object %s is corrupt\"), sha1_to_hex(d->base.sha1));\n> -               base_obj->obj = append_obj_to_pack(f, d->base.sha1,\n> +                       die(_(\"local object %s is corrupt\"), sha1_to_hex(d->sha1));\n> +               base_obj->obj = append_obj_to_pack(f, d->sha1,\n>                                         base_obj->data, base_obj->size, type);\n>                 find_unresolved_deltas(base_obj);\n>                 display_progress(progress, nr_resolved_deltas);\n> @@ -1495,7 +1554,7 @@ static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)\n>\n>  static void show_pack_info(int stat_only)\n>  {\n> -       int i, baseobjects = nr_objects - nr_deltas;\n> +       int i, baseobjects = nr_objects - nr_ref_deltas - nr_ofs_deltas;\n>         unsigned long *chain_histogram = NULL;\n>\n>         if (deepest_delta)\n> @@ -1680,11 +1739,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n>         objects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n>         if (show_stat)\n>                 obj_stat = xcalloc(nr_objects + 1, sizeof(struct object_stat));\n> -       deltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n> +       ofs_deltas = xcalloc(nr_objects, sizeof(struct ofs_delta_entry));\n>         parse_pack_objects(pack_sha1);\n>         resolve_deltas();\n>         conclude_pack(fix_thin_pack, curr_pack, pack_sha1);\n> -       free(deltas);\n> +       free(ofs_deltas);\n> +       free(ref_deltas);\n>         if (strict)\n>                 foreign_nr = check_objects();\n>\n> --\n> 2.2.0.513.g477eb31\n>\n> -- 8< --\n"},{"id":"255893","messageId":"xmqq61b9wqil.fsf@gitster.dls.corp.google.com","threadId":"38310","inReplyTo":"CAHKF-Atr_ezupL02aW08S-6NGGLi55vHuVep1mQvOaQq0Xh=FA@mail.gmail.com","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-10T18:49:38Z","receivedAt":"2015-02-10T18:49:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"matthew sporleder <msporleder@gmail.com> writes:\n\n> I'm having trouble getting this new patch to apply.\n\nApply the first one, replace all object_entry_extra with\nobject_stat, replace all objects_extra with obj_stat and amend the\nfirst one.  Then apply this one.\n"},{"id":"255927","messageId":"CAHKF-AsF=8n0zxmbYfEKnBgOAnjqW_Psw1eNsZtyMjDchyt5zA@mail.gmail.com","threadId":"38310","inReplyTo":"xmqq61b9wqil.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2015-02-11T13:01:42Z","receivedAt":"2015-02-11T13:01:42Z","isPatch":true,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"On Tue, Feb 10, 2015 at 12:49 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> matthew sporleder <msporleder@gmail.com> writes:\n>\n>> I'm having trouble getting this new patch to apply.\n>\n> Apply the first one, replace all object_entry_extra with\n> object_stat, replace all objects_extra with obj_stat and amend the\n> first one.  Then apply this one.\n\nI got this to work and had a good experience but got this from an arm user:\n\nCloning into 'NetBSD-src-git'...\nremote: Counting objects: 3484984, done.\nremote: Compressing objects: 100% (636083/636083), done.\nerror: index-pack died of signal 10\nfatal: index-pack failed\n      125.84 real         0.13 user         0.49 sys\n\nCore was generated by `git'.\nProgram terminated with signal SIGBUS, Bus error.\n#0  0x00045f88 in cmd_index_pack ()\n(gdb) bt\n#0  0x00045f88 in cmd_index_pack ()\n#1  0x00014058 in handle_builtin ()\n#2  0x00129358 in main ()\n\n\nI will wait for the \"official\" patch and ask if my friend can compile with -g.\n"},{"id":"255928","messageId":"CACsJy8CsGKK7Dt4aHHg=GQ9Y2h8bBPBVWnHwqu=CRDkYzzyGkg@mail.gmail.com","threadId":"38310","inReplyTo":"CAHKF-AsF=8n0zxmbYfEKnBgOAnjqW_Psw1eNsZtyMjDchyt5zA@mail.gmail.com","subject":"Re: [PATCH] index-pack: reduce memory footprint a bit","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-11T13:10:05Z","receivedAt":"2015-02-11T13:10:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 11, 2015 at 8:01 PM, matthew sporleder <msporleder@gmail.com> wrote:\n> On Tue, Feb 10, 2015 at 12:49 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>> matthew sporleder <msporleder@gmail.com> writes:\n>>\n>>> I'm having trouble getting this new patch to apply.\n>>\n>> Apply the first one, replace all object_entry_extra with\n>> object_stat, replace all objects_extra with obj_stat and amend the\n>> first one.  Then apply this one.\n>\n> I got this to work and had a good experience but got this from an arm user:\n>\n> Cloning into 'NetBSD-src-git'...\n> remote: Counting objects: 3484984, done.\n> remote: Compressing objects: 100% (636083/636083), done.\n> error: index-pack died of signal 10\n> fatal: index-pack failed\n>       125.84 real         0.13 user         0.49 sys\n>\n> Core was generated by `git'.\n> Program terminated with signal SIGBUS, Bus error.\n\nIt might be the effect of __attribute__((packed)). Maybe you could try\nagain without that in builtin/index-pack.c. Also could you run gdb and\ndo\n\np sizeof(*ofs_deltas)\n\n? No need to actually run it from gdb.\n-- \nDuy\n"}]}