{"thread":{"id":"24632","subject":"Git server eats all memory","startedAt":"2010-08-04T14:57:39Z","lastAt":"2018-06-20T14:54:22Z","messageCount":28,"participants":["Ivan Kanis","Matthieu Moy","Avery Pennarun","Jared Hance","Nguyen Thai Ngoc Duy","Jakub Narebski","Dmitry Potapov","Tomas Carnecky","Robin H. Johnson","Sverre Rabbelier","Sam Vilain","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"147116","messageId":"wesfwyupgrg.fsf@kanis.fr","threadId":"24632","inReplyTo":null,"subject":"Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-09@kanis.fr","sentAt":"2010-08-04T14:57:39Z","receivedAt":"2010-08-04T14:57:39Z","isPatch":false,"sender":{"key":"expire-by-2010-08-09@kanis.fr","avatar":null},"body":"Hello,\n\nI am running git 1.7.0.4 on Debian stable. I have compiled git from\nsource. I am cloning using ssh.The client and the server are running\nthe same version. The bare repository is 4.5G with various binary files,\nI have 6G of memory on my server.\n\nI am having problem with memory ballooning when receiving object\nfrom the server. The amount of memory used on the server seems to be same\nsize as the object received.\n\nI have been discussing this quite a bit on #git at freenode. Hilary said\nit was maybe due to file mmap that caused the memory reported on top to\nbe skewed.\n\nI have ran two clones side by side and noticed that the server started\nto swap and performance were awful. This means that if two developers\nare doing a clone at the same time they will notice the slowness. \n\nAnother worry is that we're planning to have more repositories of the\nsame size and the server just won't scale.\n\nI am wondering if anyone has seen this behavior? I'll do whatever I can\nto troubleshoot the problem. I know C but I just don't know where to\nlook at. Any help would be very much appreciated.\n\nKind regards,\n-- \nIvan Kanis\n\nLet a fool hold his tongue and he will pass for a sage.\n -- Publilius Syrus \n"},{"id":"147117","messageId":"vpqwrs6fk46.fsf@bauges.imag.fr","threadId":"24632","inReplyTo":"wesfwyupgrg.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-08-04T15:55:21Z","receivedAt":"2010-08-04T15:55:21Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Ivan Kanis writes:\n\n> Hello,\n>\n> I am running git 1.7.0.4 on Debian stable. I have compiled git from\n> source. I am cloning using ssh.The client and the server are running\n> the same version. The bare repository is 4.5G with various binary files,\n> I have 6G of memory on my server.\n\nI never tried that size myself, but according to what I read on the\nmailing list, that should remain within what Git could manage, maybe a\nbit painfully.\n\nThe standard followup question in your case is: is the repository\nfully packed on the server? If not, maybe \"git gc\" or \"git gc\n--aggressive\" (expansive, but a one-time operation) could help.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"147124","messageId":"87iq3qgtc9.fsf@kanis.fr","threadId":"24632","inReplyTo":"vpqwrs6fk46.fsf@bauges.imag.fr","subject":"Re: Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-09@kanis.fr","sentAt":"2010-08-04T17:50:46Z","receivedAt":"2010-08-04T17:50:46Z","isPatch":false,"sender":{"key":"expire-by-2010-08-09@kanis.fr","avatar":null},"body":"Matthieu Moy wrote:\n\n> The standard followup question in your case is: is the repository\n> fully packed on the server? If not, maybe \"git gc\" or \"git gc\n> --aggressive\" (expansive, but a one-time operation) could help.\n\nHi Mathieu,\n\nI'll give it a shot when I get back to work.\n\nI have done the following:\n\ngit repack -adf --window=100 --depth=20 --window-memory=50m\n\nIt greatly helps the compression stage that used up all the memory. The\nproblem is in the next phase when receiving objects. It is a bit strange\nas receiving objects should not take up any memory on the server.\n\nTake care,\n-- \nIvan Kanis\nhttp://kanis.fr\n\nEmail is a wonderful thing for people whose role in life is to be on\ntop of things. But not for me; my role is to be on the bottom of\nthings. What I do takes long hours of studying and uninterruptible\nconcentration.\n -- Donald Knuth \n"},{"id":"147131","messageId":"AANLkTimwy6GumHYSTo2je_hOUO80KEpx4_8z3iOoZyc0@mail.gmail.com","threadId":"24632","inReplyTo":"wesfwyupgrg.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-08-04T20:12:17Z","receivedAt":"2010-08-04T20:12:17Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Aug 4, 2010 at 10:57 AM, Ivan Kanis\n wrote:\n> I am running git 1.7.0.4 on Debian stable. I have compiled git from\n> source. I am cloning using ssh.The client and the server are running\n> the same version. The bare repository is 4.5G with various binary files,\n> I have 6G of memory on my server.\n>\n> I am having problem with memory ballooning when receiving object\n> from the server. The amount of memory used on the server seems to be same\n> size as the object received.\n\nGit works fine with huge repositories; it does not work fine at all\nwith very large individual objects in a repository, and it does what\nyou're experiencing. There are a few minor workarounds (like the\nrepack command someone mentioned) that slightly reduce the symptoms,\nbut the symptoms will crop up again eventually.\n\nThere's at least one project intended to solve this that people have linked to:\nhttp://caca.zoy.org/wiki/git-bigfiles\n\n...but it's incomplete and it doesn't look like their repo has changed\nin some time.\n\nI'm not aware of any workaround to this sort of thing other than\n\"don't store large objects in git.\" You can split the big objects\ninto a bunch of small objects, as bup does, but then you have to\nreassemble them all, which is inconvenient.\n\nSorry,\n\nAvery\n"},{"id":"147148","messageId":"87ocdhlgbl.fsf@kanis.fr","threadId":"24632","inReplyTo":"AANLkTimwy6GumHYSTo2je_hOUO80KEpx4_8z3iOoZyc0@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-10@kanis.fr","sentAt":"2010-08-05T06:33:02Z","receivedAt":"2010-08-05T06:33:02Z","isPatch":false,"sender":{"key":"expire-by-2010-08-10@kanis.fr","avatar":null},"body":"Avery Pennarun wrote:\n\n> On Wed, Aug 4, 2010 at 10:57 AM, Ivan Kanis\n\n>> I am having problem with memory ballooning when receiving object\n>> from the server. The amount of memory used on the server seems to be same\n>> size as the object received.\n\n> Git works fine with huge repositories; it does not work fine at all\n> with very large individual objects in a repository, and it does what\n> you're experiencing.\n\nHello Avery,\n\nThe largest object is 120M. I didn't describe the problem very\naccurately. The memory consumed is the *sum* of the objects\ndownloaded. \n\nFor example, a repository of 4G consumes 4G of memory at the end of the\nreceiving objects phase. What is very interesting is the total of\nobjects downloaded is the same as the memory consumed. That makes me\nthink there is a link somewhere. Surely it shouldn't consume that much\nmemory.\n\nI am ready to do whatever to diagnose the problem. I know C pretty well\nand am ready to look into it but I am not sure where to start.\n\nTake care,\n-- \nIvan Kanis\nhttp://kanis.fr\n\nSeriousness is the only refuge of the shallow.\n -- Oscar Wilde \n"},{"id":"147267","messageId":"20100805224521.GB22430@localhost.localdomain","threadId":"24632","inReplyTo":"87ocdhlgbl.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Jared Hance","fromEmail":"jaredhance@gmail.com","sentAt":"2010-08-05T22:45:22Z","receivedAt":"2010-08-05T22:45:22Z","isPatch":false,"sender":{"key":"jaredhance@gmail.com","avatar":"https://avatars.githubusercontent.com/u/170192?v=4"},"body":"On Thu, Aug 05, 2010 at 08:33:02AM +0200, Ivan Kanis wrote:\n> I am ready to do whatever to diagnose the problem. I know C pretty well\n> and am ready to look into it but I am not sure where to start.\n\nIt sounds like Git is reading each object into memory to send it to\nthe client, but doesn't remember to free the memory at the end of\nsending the object, so it remains as a memory leak.\n\nI would look in the code for malloc calls that don't have a free call,\nor spots where free calls might not be hit.\n"},{"id":"147274","messageId":"AANLkTikt7LuhxHhOqPm2P-2hzXP54YThX5FRxF4yCFZu@mail.gmail.com","threadId":"24632","inReplyTo":"87ocdhlgbl.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-06T01:37:42Z","receivedAt":"2010-08-06T01:37:42Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis\n wrote:\n> I am ready to do whatever to diagnose the problem. I know C pretty well\n> and am ready to look into it but I am not sure where to start.\n\nTry \"git pack-objects --all --stdout > /dev/null\" on the repo on\nserver to see if it uses the same amount of memory you saw in cloning.\nYou can then try debugging that command if it does.\n-- \nDuy\n"},{"id":"147275","messageId":"AANLkTi=tf51FWkZZFw9cF=pcCyadgp7a9EXK=KQ6GSQS@mail.gmail.com","threadId":"24632","inReplyTo":"AANLkTikt7LuhxHhOqPm2P-2hzXP54YThX5FRxF4yCFZu@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-06T01:51:33Z","receivedAt":"2010-08-06T01:51:33Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Aug 6, 2010 at 11:37 AM, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis\n> wrote:\n>> I am ready to do whatever to diagnose the problem. I know C pretty well\n>> and am ready to look into it but I am not sure where to start.\n>\n> Try \"git pack-objects --all --stdout > /dev/null\" on the repo on\n> server to see if it uses the same amount of memory you saw in cloning.\n> You can then try debugging that command if it does.\n\nNaah, git pack-objects needs list of commit tips. Try\ngit for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null\n-- \nDuy\n"},{"id":"147307","messageId":"m3r5icnfgk.fsf@localhost.localdomain","threadId":"24632","inReplyTo":"AANLkTi=tf51FWkZZFw9cF=pcCyadgp7a9EXK=KQ6GSQS@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-08-06T11:34:11Z","receivedAt":"2010-08-06T11:34:11Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n> On Fri, Aug 6, 2010 at 11:37 AM, Nguyen Thai Ngoc Duy wrote:\n>> On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis wrote:\n\n>>> I am ready to do whatever to diagnose the problem. I know C pretty well\n>>> and am ready to look into it but I am not sure where to start.\n>>\n>> Try \"git pack-objects --all --stdout > /dev/null\" on the repo on\n>> server to see if it uses the same amount of memory you saw in cloning.\n>> You can then try debugging that command if it does.\n> \n> Naah, git pack-objects needs list of commit tips. Try\n> git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null\n\nNitpick: git-for-each-ref has `--format' option, no need for `cut'.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"147330","messageId":"87hbj74pve.fsf@kanis.fr","threadId":"24632","inReplyTo":"AANLkTi=tf51FWkZZFw9cF=pcCyadgp7a9EXK=KQ6GSQS@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-11@kanis.fr","sentAt":"2010-08-06T17:23:17Z","receivedAt":"2010-08-06T17:23:17Z","isPatch":false,"sender":{"key":"expire-by-2010-08-11@kanis.fr","avatar":null},"body":"Nguyen Thai Ngoc Duy wrote:\n\n> Naah, git pack-objects needs list of commit tips. Try\n> git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null\n\nJared Hance wrote:\n\n> I would look in the code for malloc calls that don't have a free call,\n> or spots where free calls might not be hit.\n\nHello Jared and Nguyen,\n\nThank you Nguyen for your command. I can now reproduce the problem\nwithout needing the network. I have been following Jared lead today on\na potential memory leak. Here is what I found out.\n\nI downloaded the latest release of git 1.7.2.1 and compiled it with\ndebugging support. I ran valgrind on the command and found two memory\nleaks. I put the output at the bottom of the e-mail as it's not very\ninteresting. I patched one of the leak in pack_objects.c but got the\nsame problem: over 4G of memory consumption for a 4G repository.\n\nI've come to the conclusion that it's not a memory leak. \n\nThis afternoon I put macro around the following functions: xmalloc\nxmallocz, xrealloc, xcalloc and xmmap. It reported the line of code and\nsize passed in each functions. I then run the result through a script\nthat totaled the amount used by each bit of code.\n\nHere are the top 3 consumers:\n\n| function | source | size in M |\n|----------+----------------------------+-----------|\n| xrealloc | builtin/pack-objects.c:690 | 86 |\n| xmallocz | patch-delta.c:36 | 301 |\n| xmmap | sha1_file.c:772 | 4393 |\n\nI expected the malloc to take 4G but was surprised it didn't. It seems\nto be mmap taking all the memory. I am not familiar with that function,\nit looks like it's mapping memory to a file... Is it reasonable to mmap\nso much memory?\n\nToday I chatted with someone on freenode #git and he reported the same\nproblem on his 2G repository, I am glad I am not the only one seeing\nthis ;)\n\nI tried reading the code but it's going over my head. I'll look at is\nsome more next monday.\n\nIf anyone is familiar with the code source of git I would love to have\nsome insight into this.\n\nTake care,\n\nIvan Kanis\n\nPS: output of valgrind --leak-check=full\n\n65 bytes in 1 blocks are definitely lost in loss record 4 of 7\n at 0x4C2260E: malloc (vg_replace_malloc.c:207)\n by 0x4C22797: realloc (vg_replace_malloc.c:429)\n by 0x4C600D: xrealloc (wrapper.c:80)\n by 0x4B7939: strbuf_grow (strbuf.c:70)\n by 0x4B80BA: strbuf_addf (strbuf.c:201)\n by 0x4832EF: system_path (exec_cmd.c:37)\n by 0x483411: setup_path (exec_cmd.c:104)\n by 0x404AF2: main (git.c:536)\n\n512 bytes in 1 blocks are definitely lost in loss record 5 of 8\n at 0x4C203E4: calloc (vg_replace_malloc.c:397)\n by 0x4C5F9D: xcalloc (wrapper.c:96)\n by 0x445741: cmd_pack_objects (pack-objects.c:2117)\n by 0x4048EE: handle_internal_command (git.c:270)\n by 0x404B03: main (git.c:470)\n-- \nhttp://kanis.fr\n\nEverything should be made as simple as possible, but not simpler.\n -- Albert Einstein \n"},{"id":"147377","messageId":"AANLkTinyX9cABkEDy3HBZoDVNWos2djNBSaw2Hg_yzAO@mail.gmail.com","threadId":"24632","inReplyTo":"87hbj74pve.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2010-08-07T06:42:08Z","receivedAt":"2010-08-07T06:42:08Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:\n>\n> I expected the malloc to take 4G but was surprised it didn't. It seems\n> to be mmap taking all the memory. I am not familiar with that function,\n> it looks like it's mapping memory to a file... Is it reasonable to mmap\n> so much memory?\n\nAFAIK, Git does not need to mmap the whole pack to memory, but it\nis more efficient to mmap the whole pack wherever possible, because\nit has a completely random access, so if you store only one sliding\nwindow, you will have to re-read it many times. Besides, mmap size\ndoes not mean that so much physical memory is used. Pages should\nbe loaded when they are necessary, and if you have more than one\nclient cloning the same repo, this memory should be shared by them.\n\n\nDmitry\n"},{"id":"147491","messageId":"AANLkTinT-taMqB87moLZ_CLio2USnxsriq6U0EJvSGte@mail.gmail.com","threadId":"24632","inReplyTo":"wesbp9cnnag.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-09T09:57:00Z","receivedAt":"2010-08-09T09:57:00Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Aug 9, 2010 at 7:33 PM, Ivan Kanis\n wrote:\n> Nguyen Thai Ngoc Duy wrote:\n>\n>> Can you send me massif report of git pack-objects? It'd be interesting\n>> to see how memory is allocated.\n>\n> Hi Nguyen,\n>\n> I have attached the massif report.\n\nThanks. It does not look like it used a lot of memory (~50MB) (viewed\nwith ms_print). Git allocates 32 bytes per tree and 48 per commit plus\nall tree contents, and all that will stay in memory until the end.\n\nThis command on git already gives me ~40MB peak. Are you sure you ran\nit on your big repo?\n\necho|valgrind --tool=massif ./git pack-objects --all --stdout > /dev/null\n-- \nDuy\n"},{"id":"147493","messageId":"wesy6cgm6wd.fsf_-_@kanis.fr","threadId":"24632","inReplyTo":"AANLkTinyX9cABkEDy3HBZoDVNWos2djNBSaw2Hg_yzAO@mail.gmail.com","subject":"Excessive mmap [was Git server eats all memory]","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-14@kanis.fr","sentAt":"2010-08-09T10:12:34Z","receivedAt":"2010-08-09T10:12:34Z","isPatch":false,"sender":{"key":"expire-by-2010-08-14@kanis.fr","avatar":null},"body":"Dmitry Potapov wrote:\n\n> On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:\n>>\n>> I expected the malloc to take 4G but was surprised it didn't. It seems\n>> to be mmap taking all the memory. I am not familiar with that function,\n>> it looks like it's mapping memory to a file... Is it reasonable to mmap\n>> so much memory?\n>\n> AFAIK, Git does not need to mmap the whole pack to memory, but it\n> is more efficient to mmap the whole pack wherever possible, because\n> it has a completely random access, so if you store only one sliding\n> window, you will have to re-read it many times. Besides, mmap size\n> does not mean that so much physical memory is used. Pages should\n> be loaded when they are necessary, and if you have more than one\n> client cloning the same repo, this memory should be shared by them.\n\nI have clone identical repositories and the system starts to swap. I\nthink it shows that cloning two repository doesn't share mmap.\n\nI saw this constant defined in git-compat-util.h\n\n/* This value must be multiple of (pagesize * 2) */\n#define DEFAULT_PACKED_GIT_WINDOW_SIZE \\\n (sizeof(void*) >= 8 \\\n ? 1 * 1024 * 1024 * 1024 \\\n : 32 * 1024 * 1024)\n\nIf I read this correctly git is allocating 1G of mmap on 64 bit\narchitecture. Isn't that a bit much? I am running on a 64 bit server so\nI have bumped DEFAULT_PACKED_GIT_WINDOW_SIZE down to 64M but, alas, the\npack command still takes over 4G...\n\nI'll keep investigating today,\n\nTake care,\n-- \nIvan Kanis\n\nNothing in life is to be feared. It is only to be understood.\n -- Marie Curie \n"},{"id":"147495","messageId":"AANLkTi=6JcwLuyNWq9oYzE_E+7DSn-sEvR-X3AHvXM6U@mail.gmail.com","threadId":"24632","inReplyTo":"wesy6cgm6wd.fsf_-_@kanis.fr","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2010-08-09T12:35:11Z","receivedAt":"2010-08-09T12:35:11Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Aug 09, 2010 at 12:12:34PM +0200, Ivan Kanis wrote:\n>\n> I have clone identical repositories and the system starts to swap. I\n> think it shows that cloning two repository doesn't share mmap.\n\nThough Git uses MAP_PRIVATE with mmap, this only marks mapped pages as\ncopy-on-write. Because cloning does not change the pack file, all those\npages should be shared. So, the only reasons for swapping could be:\n- each cloning operation access to different pages at the same time, so\n it means more pages have to be loaded in memory to allow two programs\n to run simultaneously.\n- each operation allocates 387Mb (accordingly to your earlier data), so\n it may add more memory pressure.\n\n>\n> I saw this constant defined in git-compat-util.h\n>\n> /* This value must be multiple of (pagesize * 2) */\n> #define DEFAULT_PACKED_GIT_WINDOW_SIZE \\\n>        (sizeof(void*) >= 8 \\\n>                ?  1 * 1024 * 1024 * 1024 \\\n>                : 32 * 1024 * 1024)\n>\n> If I read this correctly git is allocating 1G of mmap on 64 bit\n> architecture. Isn't that a bit much?\n\nOn 64-bit architecture, you have plenty virtual space, and mapping\na file to memory should not take much physical memory (only space\nneeded for system tables). You can reduce core.packedGitWindowSize\nin the Git configuration to see if it helps, but I doubt that it\nwill have any noticeable effect.\n\n\nDmitry\n"},{"id":"147513","messageId":"westyn3n3sa.fsf@kanis.fr","threadId":"24632","inReplyTo":"AANLkTi=6JcwLuyNWq9oYzE_E+7DSn-sEvR-X3AHvXM6U@mail.gmail.com","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-14@kanis.fr","sentAt":"2010-08-09T16:34:29Z","receivedAt":"2010-08-09T16:34:29Z","isPatch":false,"sender":{"key":"expire-by-2010-08-14@kanis.fr","avatar":null},"body":"Dmitry Potapov wrote:\n\n> Though Git uses MAP_PRIVATE with mmap, this only marks mapped pages as\n> copy-on-write. Because cloning does not change the pack file, all those\n> pages should be shared.\n\nI reran the test today. One client is receiving while the other one is\ncompressing. I have to interrupt both client because the server is\nbecoming unusable. I am sure you are right about sharing the pages but I\ncan't test it.\n\n> On 64-bit architecture, you have plenty virtual space, and mapping\n> a file to memory should not take much physical memory (only space\n> needed for system tables). \n\nWhat I can tell from the mmap man page is that it should map memory to a\nfile. I assume it shouldn't take up physical memory. However I am seeing\nphysical memory being consumed. It might be a feature of the kernel. Is\nthere a way to turn it off?\n\n> You can reduce core.packedGitWindowSize in the Git configuration to\n> see if it helps, but I doubt that it will have any noticeable effect.\n\nIt was worth a shot, it didn't help...\n\nLooking some more into it today the bulk of the memory allocation\nhappens in write_pack_file in the following loop.\n\nfor (; i < nr_objects; i++) {\n if (!write_one(f, objects + i, &offset))\n break;\n display_progress(progress_state, written);\n}\n\nThis eventually calls write_object, here I am wondering if the\nunuse_pack function is doing its job. As far as I can tell it writes a\nnull in memory, that I think is not enough to reclaim memory.\n\nI also looked at the use_pack function where the mmap is\nhappening. Would it be worth refactoring this function so that it uses\nan index withing a file instead of mmap?\n\nUnless I hear of a better idea I'll be trying that tomorrow...\n\nTake care,\n-- \nIvan Kanis\n\nI can stand brute force, but brute reason is quite unbearable. There\nis something unfair about its use. It is hitting below the intellect.\n -- Oscar Wilde \n"},{"id":"147515","messageId":"AANLkTiktriuvciNTNPD4941AG3th6rWwUYT4v_UnaAz3@mail.gmail.com","threadId":"24632","inReplyTo":"westyn3n3sa.fsf@kanis.fr","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-08-09T16:50:30Z","receivedAt":"2010-08-09T16:50:30Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Aug 9, 2010 at 12:34 PM, Ivan Kanis\n wrote:\n> Dmitry Potapov wrote:\n>> On 64-bit architecture, you have plenty virtual space, and mapping\n>> a file to memory should not take much physical memory (only space\n>> needed for system tables).\n>\n> What I can tell from the mmap man page is that it should map memory to a\n> file. I assume it shouldn't take up physical memory. However I am seeing\n> physical memory being consumed. It might be a feature of the kernel. Is\n> there a way to turn it off?\n\n'ps axu' will show two columns: VSIZE and RSS. The only one that\nactually matters is RSS.\n\nWhen you mmap a file, it will immediately consume a lot of VSIZE - but\nthis won't affect your available system memory, because you have only\nconsumed \"virtual\" memory. Instead of swapping that memory out to the\nswap file, the kernel knows that this chunk of virtual memory is\nalready on disk - inside the mmap'd file.\n\nWhen you access some of the pages of the mmap'd file, the kernel will\nswap those pages into memory, which increases RSS. This uses *real*\nmemory on the system.\n\nAs git generates a new pack file, it needs to access every single page\nof every single pack that it's reading from, so eventually, all the\nstuff you need will get sucked into RSS, so you'll see that number\ngrow and grow. If your packfiles are huge, this is a lot of memory.\n\nNow, the kernel is supposed to be smart enough to release old pages\nout of RSS if you stop using them; it's no different from what the\nkernel does with any cached file data. So it shouldn't be expensive\nto mmap instead of just reading the file.\n\n> Looking some more into it today the bulk of the memory allocation\n> happens in write_pack_file in the following loop.\n>\n> for (; i < nr_objects; i++) {\n>    if (!write_one(f, objects + i, &offset))\n>        break;\n>    display_progress(progress_state, written);\n> }\n>\n> This eventually calls write_object, here I am wondering if the\n> unuse_pack function is doing its job. As far as I can tell it writes a\n> null in memory, that I think is not enough to reclaim memory.\n\nWhat do you mean by the \"memory allocation\" happens here? How are you\nmeasuring it?\n\nunuse_pack indeed doesn't free any memory; it just zeroes a pointer\nand decreases a refcount. I don't know much about this code, but I\nassume something else goes and cleans up the mmaps later.\n\nIn any case, mmap/munmap have little to do with your \"real\" memory\nusage. munmap() won't free any actual kernel memory; the used pages\nwill still be floating around in disk cache.\n\n> I also looked at the use_pack function where the mmap is\n> happening. Would it be worth refactoring this function so that it uses\n> an index withing a file instead of mmap?\n>\n> Unless I hear of a better idea I'll be trying that tomorrow...\n\nI wouldn't expect this to help, but I would be interested to hear if it does.\n\nIf the problem is simply that you're flooding the kernel disk cache\nwith data you'll use only once, to the detriment of everything else on\nthe system, then one thing that might help could be posix_fadvise:\n\n posix_fadvise(fd, ofs, len, POSIX_FADV_DONTNEED);\n\nbup uses this when backing up huge files, since it knows it's only\ngoing to use each block once, and this seemed to decrease system load\n(without affecting bup's own performance) in some test cases.\nHowever, it uses this for filesystem files, not packs, so it's a\ndifferent use case.\n\nOn the other hand, perhaps a more important question is: why does git\nfeel like it needs to generate entirely new packs for each person\ndoing a clone on your system? Shouldn't it be reusing existing ones\nand just streaming them straight out to the recipient?\n\nHave fun,\n\nAvery\n"},{"id":"147523","messageId":"871va7lm9t.fsf@kanis.fr","threadId":"24632","inReplyTo":"AANLkTinT-taMqB87moLZ_CLio2USnxsriq6U0EJvSGte@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-14@kanis.fr","sentAt":"2010-08-09T17:38:06Z","receivedAt":"2010-08-09T17:38:06Z","isPatch":false,"sender":{"key":"expire-by-2010-08-14@kanis.fr","avatar":null},"body":"Nguyen Thai Ngoc Duy wrote:\n\n> This command on git already gives me ~40MB peak. Are you sure you ran\n> it on your big repo?\n\nHi Nguyen,\n\nYes I ran it on the big repo...\n\nTake care,\n-- \nIvan Kanis\nhttp://kanis.fr\n\nMen of lofty genius when they are doing the least work are the most\nactive.\n -- Leonardo da Vinci \n"},{"id":"147524","messageId":"4C603EAF.10500@dbservice.com","threadId":"24632","inReplyTo":"AANLkTiktriuvciNTNPD4941AG3th6rWwUYT4v_UnaAz3@mail.gmail.com","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Tomas Carnecky","fromEmail":"tom@dbservice.com","sentAt":"2010-08-09T17:45:19Z","receivedAt":"2010-08-09T17:45:19Z","isPatch":false,"sender":{"key":"tom@dbservice.com","avatar":"https://gravatar.com/avatar/900a300bdd1a8bbe086008ad78210bbee2ad2803b7d50a5cba04c1e9404bd6d2?d=mp&s=160"},"body":"On 8/9/10 6:50 PM, Avery Pennarun wrote:\n> On the other hand, perhaps a more important question is: why does git\n> feel like it needs to generate entirely new packs for each person\n> doing a clone on your system? Shouldn't it be reusing existing ones\n> and just streaming them straight out to the recipient?\n\nIsn't that something that the rev cache is supposed to fix?\n\ntom\n"},{"id":"147527","messageId":"AANLkTimjE5qF4cTRY=63OY5WdZhp6LqGbLDhbm1hXExX@mail.gmail.com","threadId":"24632","inReplyTo":"4C603EAF.10500@dbservice.com","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-08-09T18:17:27Z","receivedAt":"2010-08-09T18:17:27Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Aug 9, 2010 at 1:45 PM, Tomas Carnecky wrote:\n> On 8/9/10 6:50 PM, Avery Pennarun wrote:\n>> On the other hand, perhaps a more important question is: why does git\n>> feel like it needs to generate entirely new packs for each person\n>> doing a clone on your system?  Shouldn't it be reusing existing ones\n>> and just streaming them straight out to the recipient?\n>\n> Isn't that something that the rev cache is supposed to fix?\n\nI wouldn't think so - at least not when cloning a repository from\nscratch. The whole idea of a clone is you ought to be able to copy\nthe pack contents verbatim since you should want \"all\" the objects.\nThough maybe I've missed something... I've never read the code for\nthat stuff.\n\nHave fun,\n\nAvery\n"},{"id":"147562","messageId":"AANLkTinx+NXMBY3K-KGtHQA1=pdz66o5bRgeG8U_Kyd7@mail.gmail.com","threadId":"24632","inReplyTo":"AANLkTiktriuvciNTNPD4941AG3th6rWwUYT4v_UnaAz3@mail.gmail.com","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2010-08-09T21:28:17Z","receivedAt":"2010-08-09T21:28:17Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Aug 09, 2010 at 12:50:30PM -0400, Avery Pennarun wrote:\n>\n> On the other hand, perhaps a more important question is: why does git\n> feel like it needs to generate entirely new packs for each person\n> doing a clone on your system?  Shouldn't it be reusing existing ones\n> and just streaming them straight out to the recipient?\n\nGit cannot send the whole pack, in general, because it sends only those\nobjects that are requested by the client. So, except the initial cloning,\nit is never the whole pack. Moreover, even during the initial cloning,\nit may not be the whole, but only visible objects from it. So, git has\nto generate a new pack for every clone operation. But the generated pack\nshould reuse deltas from the initial pack, so it should not a very\nexpensive operation.\n\n\nDmitry\n"},{"id":"147600","messageId":"robbat2-20100810T004153-279468425Z@orbis-terrarum.net","threadId":"24632","inReplyTo":"wesfwyupgrg.fsf@kanis.fr","subject":"Re: Git server eats all memory","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2010-08-10T00:46:36Z","receivedAt":"2010-08-10T00:46:36Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"On Wed, Aug 04, 2010 at 04:57:39PM +0200, Ivan Kanis wrote:\n> I am wondering if anyone has seen this behavior? I'll do whatever I can\n> to troubleshoot the problem. I know C but I just don't know where to\n> look at. Any help would be very much appreciated.\nWe've seen a similar problem in experimental planning for migrating the\ncore Gentoo repository to Git. That's a 900MiB packfile ('git repack\n-adf --window=250 --depth=250' taking 2 hours).\n\nMultiple concurrent full clones push the server into swap. We had 16GiB\nof RAM, and this was still occurring.\n\nOur temporary solution plan is via hooks, if you're asking for a item\nbefore a certain point, throw an error telling you to download a\ngit-bundle from a given URL instead (as a bonus you can resume that\ntrivially).\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Developer, Trustee & Infrastructure Lead\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85\n"},{"id":"147601","messageId":"AANLkTi=DcuHEkJf1s18LUHrA2L6jUiadhxKqQBO4QCHh@mail.gmail.com","threadId":"24632","inReplyTo":"robbat2-20100810T004153-279468425Z@orbis-terrarum.net","subject":"Re: Git server eats all memory","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-08-10T02:31:24Z","receivedAt":"2010-08-10T02:31:24Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n[please don't cull the cc list]\n\nOn Mon, Aug 9, 2010 at 19:46, Robin H. Johnson wrote:\n> Our temporary solution plan is via hooks, if you're asking for a item\n> before a certain point, throw an error telling you to download a\n> git-bundle from a given URL instead (as a bonus you can resume that\n> trivially).\n\nSeems like there should be a way to tell the git server that certain\npack files should be sent to the client verbatim. Perhaps the protocol\ncould learn a new capability to support such a negotiation in which\nthe server will assume that the client either has the required packs,\nor continue negotiation under the assumption that they will be\ndownloaded first.\n\nSounds like Sam might have some relevant to say about this? Or perhaps\nthe pack caching gsoc project is relevant?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"147752","messageId":"1281522626.2512.10.camel@arcturus","threadId":"24632","inReplyTo":"AANLkTi=DcuHEkJf1s18LUHrA2L6jUiadhxKqQBO4QCHh@mail.gmail.com","subject":"Re: Git server eats all memory","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-08-11T10:30:26Z","receivedAt":"2010-08-11T10:30:26Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Mon, 2010-08-09 at 21:31 -0500, Sverre Rabbelier wrote:\n> On Mon, Aug 9, 2010 at 19:46, Robin H. Johnson wrote:\n> > Our temporary solution plan is via hooks, if you're asking for a item\n> > before a certain point, throw an error telling you to download a\n> > git-bundle from a given URL instead (as a bonus you can resume that\n> > trivially).\n> \n> Seems like there should be a way to tell the git server that certain\n> pack files should be sent to the client verbatim. Perhaps the protocol\n> could learn a new capability to support such a negotiation in which\n> the server will assume that the client either has the required packs,\n> or continue negotiation under the assumption that they will be\n> downloaded first.\n> \n> Sounds like Sam might have some relevant to say about this? Or perhaps\n> the pack caching gsoc project is relevant?\n\nSure, well the project was supposed to be primarily useful for this use\ncase. It just needs someone to pick it up and revitalize it so it can\nbe merged... looks like there's a topgit series at github.com/sirnot/git\nfrom Oct. last year.\n\nSam\n"},{"id":"147788","messageId":"weshbj1xiav.fsf@kanis.fr","threadId":"24632","inReplyTo":"AANLkTiktriuvciNTNPD4941AG3th6rWwUYT4v_UnaAz3@mail.gmail.com","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-16@kanis.fr","sentAt":"2010-08-11T15:47:36Z","receivedAt":"2010-08-11T15:47:36Z","isPatch":false,"sender":{"key":"expire-by-2010-08-16@kanis.fr","avatar":null},"body":"Hi Avery,\n\nAvery Pennarun wrote:\n\n> ... When you access some of the pages of the mmap'd file, the kernel\n> will swap those pages into memory, which increases RSS. This uses\n> *real* memory on the system...\n\nThanks for the very clear explanations\n\n> Now, the kernel is supposed to be smart enough to release old pages\n> out of RSS if you stop using them; it's no different from what the\n> kernel does with any cached file data. So it shouldn't be expensive\n> to mmap instead of just reading the file.\n\nHow can the kernel release old pages? There does not seem to be anyway\nto tell it that it doesn't need a given memory block.\n\n>> Looking some more into it today the bulk of the memory allocation\n>> happens in write_pack_file in the following loop.\n>>\n>> for (; i < nr_objects; i++) {\n>>    if (!write_one(f, objects + i, &offset))\n>>        break;\n>>    display_progress(progress_state, written);\n>> }\n>>\n>> This eventually calls write_object, here I am wondering if the\n>> unuse_pack function is doing its job. As far as I can tell it writes a\n>> null in memory, that I think is not enough to reclaim memory.\n>\n> What do you mean by the \"memory allocation\" happens here? How are you\n> measuring it?\n\nI run top and look at the RES column. I put a printf before and after\nthe code block and watch the memory go up and up.\n\n>> I also looked at the use_pack function where the mmap is\n>> happening. Would it be worth refactoring this function so that it uses\n>> an index withing a file instead of mmap?\n>>\n>> Unless I hear of a better idea I'll be trying that tomorrow...\n>\n> I wouldn't expect this to help, but I would be interested to hear if\n> it does.\n\nI got caught up with other thing at work but I think I'll be able to try\nFriday.\n\n> If the problem is simply that you're flooding the kernel disk cache\n> with data you'll use only once, to the detriment of everything else on\n> the system, then one thing that might help could be posix_fadvise:\n>\n> posix_fadvise(fd, ofs, len, POSIX_FADV_DONTNEED);\n\nSounds interesting, I'll try sticking that in the unuse_pack function\nFriday.\n\n> On the other hand, perhaps a more important question is: why does git\n> feel like it needs to generate entirely new packs for each person\n> doing a clone on your system? Shouldn't it be reusing existing ones\n> and just streaming them straight out to the recipient?\n\nAh interesting point. Two things make me suspect the mmap is not shared\nbetween processes. One is that mmap is done with the MAP_PRIVATE flag\nwhich according to the man page doesn't share between processes. The\nsecond is that the mmap is done on a temporary file created by\nodb_mkstemp, I don't believe the name is identical between the two\nprocesses.\n\nTake care,\n-- \nIvan Kanis\nhttp://kanis.fr\n\nNobody ever went broke underestimating the intelligence of the\nAmerican public.\n -- H L Mencken \n"},{"id":"147789","messageId":"wesd3tpxhyt.fsf@kanis.fr","threadId":"24632","inReplyTo":"robbat2-20100810T004153-279468425Z@orbis-terrarum.net","subject":"Re: Git server eats all memory","fromName":"Ivan Kanis","fromEmail":"expire-by-2010-08-16@kanis.fr","sentAt":"2010-08-11T15:54:50Z","receivedAt":"2010-08-11T15:54:50Z","isPatch":false,"sender":{"key":"expire-by-2010-08-16@kanis.fr","avatar":null},"body":"Hi Robin,\n\n\"Robin H. Johnson\" wrote:\n\n> On Wed, Aug 04, 2010 at 04:57:39PM +0200, Ivan Kanis wrote:\n>> I am wondering if anyone has seen this behavior? I'll do whatever I can\n>> to troubleshoot the problem. I know C but I just don't know where to\n>> look at. Any help would be very much appreciated.\n>\n> Multiple concurrent full clones push the server into swap. We had 16GiB\n> of RAM, and this was still occurring.\n\nGlad I am not the only one seeing this :D\n\n> Our temporary solution plan is via hooks, if you're asking for a item\n> before a certain point, throw an error telling you to download a\n> git-bundle from a given URL instead (as a bonus you can resume that\n> trivially).\n\nI don't understand the solution... Do you have snippet of the hook?\n\nTake care,\n-- \nIvan Kanis\nhttp://kanis.fr\n"},{"id":"147795","messageId":"AANLkTimi8iS-vdO+=WHEGLQTQuBi75nVUM_eO=oYV+GU@mail.gmail.com","threadId":"24632","inReplyTo":"weshbj1xiav.fsf@kanis.fr","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-08-11T16:35:10Z","receivedAt":"2010-08-11T16:35:10Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Aug 11, 2010 at 11:47 AM, Ivan Kanis\n wrote:\n> Avery Pennarun wrote:\n>> Now, the kernel is supposed to be smart enough to release old pages\n>> out of RSS if you stop using them; it's no different from what the\n>> kernel does with any cached file data.  So it shouldn't be expensive\n>> to mmap instead of just reading the file.\n>\n> How can the kernel release old pages? There does not seem to be anyway\n> to tell it that it doesn't need a given memory block.\n\nThe kernel doesn't care whether you \"need\" it; it swaps out \"needed\"\npages all the time.\n\nWith a normal dirty memory page (allocated with malloc() or whatever),\nthe kernel will need to write it out to the swap file before it drops\nit from RSS. Then if the process needs to read/write that page in the\nfuture, it'll have to read it back in from the swap file and increase\nRSS again before it can be used.\n\nWith mmap'd files it's slightly different. As long as the page hasn't\nbeen modified (and as far as I know, git never writes to pages of\npackfiles) then we already know that page is safe on disk. So if the\nkernel needs to \"swap it out\", it just drops it immediately from RSS\nand doesn't do any I/O. When/if the process needs to read/write the\npage in the future, the kernel can swap it in the way it did in the\nfirst place: from the original file.\n\nIf I understand correctly, all this means that the kernel on average\ntries to drop mmap'd file pages out of RSS more than other kinds of\ndirty pages, because swapping out mmap'd pages is cheaper.\n\nIf you think about it, if you do 'cat filename' in a loop, every new\n'cat' process needs to load filename into memory. Of course the kernel\ndoesn't throw away the pages just because cat exits; it keeps a cache\nof the file's pages in memory, and just feeds them to the next 'cat'\nprocess when it starts. So the kernel keeps stuff in memory even if\nnobody is currently using it. The surprising thing (at first) is that\nthe kernel is also happy to throw away pages even if you *are* using\nthem, as long as it can get them back.\n\nSwapping is based on how frequently a page is used, not whether that\npage is currently mapped into someone's address space. (Disclaimer: I\nhaven't read the code. Maybe it does give higher priority to pages\nthat are currently mapped.)\n\n>>> Looking some more into it today the bulk of the memory allocation\n>>> happens in write_pack_file in the following loop.\n>>>\n>>> for (; i < nr_objects; i++) {\n>>>    if (!write_one(f, objects + i, &offset))\n>>>        break;\n>>>    display_progress(progress_state, written);\n>>> }\n>>>\n>>> This eventually calls write_object, here I am wondering if the\n>>> unuse_pack function is doing its job. As far as I can tell it writes a\n>>> null in memory, that I think is not enough to reclaim memory.\n>>\n>> What do you mean by the \"memory allocation\" happens here?  How are you\n>> measuring it?\n>\n> I run top and look at the RES column. I put a printf before and after\n> the code block and watch the memory go up and up.\n\nYeah, that's not a very good way to do it. The problem is that RSS is\n*guaranteed* to go up in this location: you've just accessed an mmap'd\npage you haven't used before. That's not a bug. Furthermore, if\nmultiple processes are mmap'ing the same pages, *all* those processes\nmight see their RSS go up, but it's the \"same\" pages, so that's not\nactually taking twice the physical memory.\n\nUnfortunately there are no really reliable ways to track this kind of\nmemory usage (as far as I know). The tricks I often use are:\n\n1) while sleep 1; do free; done\n\n2) vmstat 1\n\nCommand #1 will show you what's happening to your physical RAM. If\nyou run one git-repack, do you see the 'free' column decreasing by the\nsame amount as the RSS increases? If you run two repacks at once,\ndoes it increase as the sum of the two RSS columns, or just one of\nthem, or something else?\n\nCommand #2 will show you your blocks swapped in and out per second.\nThe interesting columns are si/so/bi/bo.\n\n>> On the other hand, perhaps a more important question is: why does git\n>> feel like it needs to generate entirely new packs for each person\n>> doing a clone on your system?  Shouldn't it be reusing existing ones\n>> and just streaming them straight out to the recipient?\n>\n> Ah interesting point. Two things make me suspect the mmap is not shared\n> between processes. One is that mmap is done with the MAP_PRIVATE flag\n> which according to the man page doesn't share between processes. The\n> second is that the mmap is done on a temporary file created by\n> odb_mkstemp, I don't believe the name is identical between the two\n> processes.\n\nMAP_PRIVATE is a little more complicated than that. What it means is\nthat if one of the processes *writes* to one of the pages, the other\nprocess won't see the changes. But if nobody writes to the pages -\nand I'm pretty sure nobody does - then the kernel won't just copy the\ndata for no reason, because it would be pointlessly inefficient.\n\nThat said, you're obviously experiencing bad behaviour, ie. it's not\nworking like it's supposed to, one way or another. So you shouldn't\ntrust that your kernel, or git, or even my explanations are correct :)\n\nHave fun,\n\nAvery\n"},{"id":"148292","messageId":"AANLkTimDx1Vg6kUaHgap6+khTmJYsxdiSDB6cb88-fnN@mail.gmail.com","threadId":"24632","inReplyTo":"wes4oetv31i.fsf@kanis.fr","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2010-08-17T17:07:29Z","receivedAt":"2010-08-17T17:07:29Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"Hi Ivan,\n\nOn Tue, Aug 17, 2010 at 02:26:01PM +0200, Ivan Kanis wrote:\n>\n> I have ran the following command for my tests: vmstat -SM -n 60\n>\n> Here's the stat for one git clone, I see that 4237M of memory is\n> consumed. That is roughly the size of the repository. This raises my\n> first question: why is the memory not reclaimed, at the end of the run?\n\nIt is consumed by system cache to hold the last read data, and it will\nhold it as long as there is enough free memory. When there is not enough\nfree memory, the system will free some old (long time unused) pages from\nthe system cache. This allows the system to avoid re-reading same files\nwhen there is enough memory to keep them.\n\n>\n> ls -lh objects/pack/*.pack\n> 4.2G objects/pack/pack-55ad6d01f37427ca69e6267b0cd4e5257e57272c.pack\n>\n> Is it a sensible behavior to leave a 4G file lying around?\n\nI am not sure I understand your question. This pack contains all data\nof your repository. So what did you expect to happen?\n\n> Does it get erased when people are pushing changes in?\n\nNo, usually, push adds a new pack. However, when there are too many\npacks, the garbage collector will try to repack everything in one new\npack. You can create an empty file with the same name but with .keep\nextension to preserve the specified pack from being repacked.\n\n\nDmitry\n"},{"id":"350563","messageId":"CACsJy8AeFfi_k9vMm71BTTF-3GQMFPOTXMhfwqKWT-7RFVjoNw@mail.gmail.com","threadId":"24632","inReplyTo":"wesy6cgm6wd.fsf_-_@kanis.fr","subject":"Re: Excessive mmap [was Git server eats all memory]","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-20T14:53:51Z","receivedAt":"2018-06-20T14:54:22Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jun 19, 2018 at 10:27 PM Ivan Kanis\n wrote:\n>\n> Dmitry Potapov wrote:\n>\n> > On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:\n> >>\n> >> I expected the malloc to take 4G but was surprised it didn't. It seems\n> >> to be mmap taking all the memory. I am not familiar with that function,\n> >> it looks like it's mapping memory to a file... Is it reasonable to mmap\n> >> so much memory?\n> >\n> > AFAIK, Git does not need to mmap the whole pack to memory, but it\n> > is more efficient to mmap the whole pack wherever possible, because\n> > it has a completely random access, so if you store only one sliding\n> > window, you will have to re-read it many times. Besides, mmap size\n> > does not mean that so much physical memory is used. Pages should\n> > be loaded when they are necessary, and if you have more than one\n> > client cloning the same repo, this memory should be shared by them.\n>\n> I have clone identical repositories and the system starts to swap. I\n> think it shows that cloning two repository doesn't share mmap.\n\nI doubt it (assuming you're on linux). If you suspect this, configure\ncore.packedGitWindowSize to reduce the mmap size. There are lots of\nother things in a cloning process that do not share (is this client or\nserver btw?) and things could add up.\n--\nDuy\n"}]}