{"thread":{"id":"26964","subject":"Git exhausts memory.","startedAt":"2011-04-02T05:01:59Z","lastAt":"2011-04-06T16:33:39Z","messageCount":19,"participants":["Alif Wahid","Nicolas Pitre","Nguyen Thai Ngoc Duy","David Fries","Holger Hellmuth","Shawn Pearce","Junio C Hamano","Jay Soffian"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"164944","messageId":"BANLkTin=yUtzbZjs_92FHDfs62VFFuLHwg@mail.gmail.com","threadId":"26964","inReplyTo":null,"subject":"Git exhausts memory.","fromName":"Alif Wahid","fromEmail":"alif.wahid@gmail.com","sentAt":"2011-04-02T05:01:59Z","receivedAt":"2011-04-02T05:01:59Z","isPatch":false,"sender":{"key":"alif.wahid@gmail.com","avatar":null},"body":"Hi there,\n\nI'm using Git v1.7.1 on Ubuntu v10.10 and unfortunately Git seems to\nregularly exhaust the memory on my machine and fails to compress loose\nobjects and/or collect garbage.\n\nMy Intel based dual-core machine has 2 GB of RAM and 4 GB of swap\nspace. I need to track a working tree with a handful of really large\ntarballs that rarely change and loads of really small text files that\nchange frequently. What I'm seeing is that over time whenever \"git gc\"\nruns automatically it fails with the message \"fatal: Out of memory,\nmalloc failed\". So I've been trying to manually run \"git repack -ad\n--window-memory=1g --max-pack-size=1g\" in the hope that Git will not\nexceed the physical memory. But I still get the same error message :(\n\nAs I can't make my repository public, I've attached a simple Python\nscript that generates a ~1.3 GB file containing random integers (takes\nroughly 10 min. on my machine). Then I run the following four commands\nand get the out-of-memory failure from \"git repack\". This is\neffectively emulating the scenario I have with my repository.\n\n$ git init\n$ git add ./test_data.dat\n$ git commit ./test_data.dat -m \"Test data.\"\n$ git repack -ad --window-memory=1g --max-pack-size=1g\nCounting objects: 3, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (2/2), done.\nfatal: Out of memory, malloc failed\n\nI can't find anything on the wiki about out-of-memory failures. Any\ninfo/help would be much appreciated.\n\nRegards\n\nAlif\n\n\n#! /usr/bin/env python\n\nimport sys, os, random\n\nif __name__ == '__main__':\n\n fbuff = 2**28 # file buffer size in bytes\n fints = 10**8 # number of integers to write\n fsize = 12 # number of digits to write per integer\n \n fname = os.path.splitext(os.path.basename(sys.argv[0]))[0]\n try:\n os.mkdir(fname)\n except OSError:\n pass # directory already exists\n \n fpath = os.path.join(fname, fname+'.dat')\n fhand = open(fpath, 'w', fbuff)\n \n sys.stdout.write('Writing %d MB to file \\'%s\\'\\n' % (fints*(fsize+1)/10**6, fpath))\n sys.stdout.write('This will take some time, please wait.\\n')\n sys.stdout.write('Progress:')\n sys.stdout.flush()\n\n random.seed(-1)\n fform = '%'+str(fsize)+'d\\n'\n for x in xrange(fints): \n fhand.write(fform % (random.randint(0,fints)))\n if x % (fints/10) == 0 and x > 0:\n sys.stdout.write('%3d/10' % (x/(fints/10))) \n sys.stdout.flush()\n\n sys.stdout.write(' 10/10\\n')\n sys.stdout.flush()\n fhand.close()\n\n"},{"id":"164951","messageId":"alpine.LFD.2.00.1104021103130.28032@xanadu.home","threadId":"26964","inReplyTo":"BANLkTin=yUtzbZjs_92FHDfs62VFFuLHwg@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-04-02T15:05:38Z","receivedAt":"2011-04-02T15:05:38Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 2 Apr 2011, Alif Wahid wrote:\n\n> Hi there,\n> \n> I'm using Git v1.7.1 on Ubuntu v10.10 and unfortunately Git seems to\n> regularly exhaust the memory on my machine and fails to compress loose\n> objects and/or collect garbage.\n> \n> My Intel based dual-core machine has 2 GB of RAM and 4 GB of swap\n> space. I need to track a working tree with a handful of really large\n> tarballs that rarely change and loads of really small text files that\n> change frequently. What I'm seeing is that over time whenever \"git gc\"\n> runs automatically it fails with the message \"fatal: Out of memory,\n> malloc failed\". So I've been trying to manually run \"git repack -ad\n> --window-memory=1g --max-pack-size=1g\" in the hope that Git will not\n> exceed the physical memory. But I still get the same error message :(\n\nDon't use --max-pack-size. That won't help here.\n\nHow large are those tar files?\n\n\nNicolas\n"},{"id":"164988","messageId":"BANLkTikRGQ45xvWvisMhXi4Hu2r_0GS=Gg@mail.gmail.com","threadId":"26964","inReplyTo":"alpine.LFD.2.00.1104021103130.28032@xanadu.home","subject":"Re: Git exhausts memory.","fromName":"Alif Wahid","fromEmail":"alif.wahid@gmail.com","sentAt":"2011-04-03T09:15:47Z","receivedAt":"2011-04-03T09:15:47Z","isPatch":false,"sender":{"key":"alif.wahid@gmail.com","avatar":null},"body":"Hi Nicolas,\n\nOn 3 April 2011 02:05, Nicolas Pitre wrote:\n\n> Don't use --max-pack-size.  That won't help here.\n\nI've tried only --window-memory with different values and they all\nfailed. It seems to me as though this option is simply ignored or\nnon-existent.\n\n> How large are those tar files?\n\nThe tar files aggregate to just under 2 GB and my complete working\ntree is around 3 GB. Whenever I run git-gc or git-repack they seem to\nreach a virtual mem. footprint of roughly 2.5 GB over the course of 10\nminutes and then fail with the out-of-memory error.\n\nAlif\n"},{"id":"164998","messageId":"alpine.LFD.2.00.1104031110150.28032@xanadu.home","threadId":"26964","inReplyTo":"BANLkTikRGQ45xvWvisMhXi4Hu2r_0GS=Gg@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-04-03T15:18:47Z","receivedAt":"2011-04-03T15:18:47Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 3 Apr 2011, Alif Wahid wrote:\n\n> Hi Nicolas,\n> \n> On 3 April 2011 02:05, Nicolas Pitre wrote:\n> \n> > Don't use --max-pack-size.  That won't help here.\n> \n> I've tried only --window-memory with different values and they all\n> failed. It seems to me as though this option is simply ignored or\n> non-existent.\n\nIt is not ignored, but there are situations where there are problem \nmaking it effective, especially if a few files are very large.\n\n> > How large are those tar files?\n> \n> The tar files aggregate to just under 2 GB and my complete working\n> tree is around 3 GB.\n\nWhat about the individual size for those files?\n\nSomething you can try is to simply tell Git not to attempt any delta \ncompression on those tar files using gitattributes (see the man page of \nthe same name).\n\n\nNicolas\n"},{"id":"165109","messageId":"BANLkTinCwZG3+0Ss8o9ODptg=L8LKKN7aQ@mail.gmail.com","threadId":"26964","inReplyTo":"alpine.LFD.2.00.1104031110150.28032@xanadu.home","subject":"Re: Git exhausts memory.","fromName":"Alif Wahid","fromEmail":"alif.wahid@gmail.com","sentAt":"2011-04-04T12:52:52Z","receivedAt":"2011-04-04T12:52:52Z","isPatch":false,"sender":{"key":"alif.wahid@gmail.com","avatar":null},"body":"Hi Nicolas,\n\nOn 4 April 2011 01:18, Nicolas Pitre wrote:\n>\n> Something you can try is to simply tell Git not to attempt any delta\n> compression on those tar files using gitattributes (see the man page of\n> the same name).\n>\n\nSeems to have worked. Both git-gc and git-repack appear to be less\nmemory hungry now and do actually run to completion without failure.\n\nThanks for your help.\n\nCheers\n\nAlif\n"},{"id":"165114","messageId":"BANLkTinU7x16yp+y-HW9UhkVn9SftOJCcA@mail.gmail.com","threadId":"26964","inReplyTo":"BANLkTinCwZG3+0Ss8o9ODptg=L8LKKN7aQ@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-04-04T14:57:01Z","receivedAt":"2011-04-04T14:57:01Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 4, 2011 at 7:52 PM, Alif Wahid wrote:\n> Hi Nicolas,\n>\n> On 4 April 2011 01:18, Nicolas Pitre wrote:\n>>\n>> Something you can try is to simply tell Git not to attempt any delta\n>> compression on those tar files using gitattributes (see the man page of\n>> the same name).\n\nShould we change the default to not delta if a blob exceeds predefined\nlimit (say 128M)? People who deliberately wants to delta them can\nstill set delta attr. 1.8.0 material maybe?\n\n> Seems to have worked. Both git-gc and git-repack appear to be less\n> memory hungry now and do actually run to completion without failure.\n>\n> Thanks for your help.\n-- \nDuy\n"},{"id":"165161","messageId":"20110405022235.GA4414@spacedout.fries.net","threadId":"26964","inReplyTo":"BANLkTinU7x16yp+y-HW9UhkVn9SftOJCcA@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"David Fries","fromEmail":"david@fries.net","sentAt":"2011-04-05T02:22:35Z","receivedAt":"2011-04-05T02:22:35Z","isPatch":false,"sender":{"key":"david@fries.net","avatar":"https://avatars.githubusercontent.com/u/2767875?v=4"},"body":"On Mon, Apr 04, 2011 at 09:57:01PM +0700, Nguyen Thai Ngoc Duy wrote:\n> On Mon, Apr 4, 2011 at 7:52 PM, Alif Wahid wrote:\n> > Hi Nicolas,\n> >\n> > On 4 April 2011 01:18, Nicolas Pitre wrote:\n> >>\n> >> Something you can try is to simply tell Git not to attempt any delta\n> >> compression on those tar files using gitattributes (see the man page of\n> >> the same name).\n> \n> Should we change the default to not delta if a blob exceeds predefined\n> limit (say 128M)? People who deliberately wants to delta them can\n> still set delta attr. 1.8.0 material maybe?\n\nI think it would be better to define it in terms of available memory.\nSomething like the minimum of system memory or address space, and\ndelta up to X amount of that (it might be good to leave off swap to\nreduce trashing). There has to be a better way than a straight 128MB\ndefault.\n\nThe number which works on my 8GB desktop is going to kill the computer\nin the trunk of my car with 48MB of ram. I've actually seen a 700 MB\nrepository fail with `git-gc --aggressive` on a system with 4GB of ram\nbecause it ran out of memory, it only worked by leaving off the\n--aggressive option.\n\n> > Seems to have worked. Both git-gc and git-repack appear to be less\n> > memory hungry now and do actually run to completion without failure.\n> >\n> > Thanks for your help.\n> -- \n> Duy\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n\n-- \nDavid Fries PGP pub CB1EE8F0\nhttp://fries.net/~david/\n"},{"id":"165163","messageId":"BANLkTik6XoWaehp7=5fxwcyhkTHWCO8-Eg@mail.gmail.com","threadId":"26964","inReplyTo":"20110405022235.GA4414@spacedout.fries.net","subject":"Re: Git exhausts memory.","fromName":"Alif Wahid","fromEmail":"alif.wahid@gmail.com","sentAt":"2011-04-05T04:35:56Z","receivedAt":"2011-04-05T04:35:56Z","isPatch":false,"sender":{"key":"alif.wahid@gmail.com","avatar":null},"body":"Hi everyone,\n\nOn 5 April 2011 12:22, David Fries wrote:\n> On Mon, Apr 04, 2011 at 09:57:01PM +0700, Nguyen Thai Ngoc Duy wrote:\n>> On Mon, Apr 4, 2011 at 7:52 PM, Alif Wahid wrote:\n>> > Hi Nicolas,\n>> >\n>> > On 4 April 2011 01:18, Nicolas Pitre wrote:\n>> >>\n>> >> Something you can try is to simply tell Git not to attempt any delta\n>> >> compression on those tar files using gitattributes (see the man page of\n>> >> the same name).\n>>\n>> Should we change the default to not delta if a blob exceeds predefined\n>> limit (say 128M)? People who deliberately wants to delta them can\n>> still set delta attr. 1.8.0 material maybe?\n>\n> I think it would be better to define it in terms of available memory.\n> Something like the minimum of system memory or address space, and\n> delta up to X amount of that (it might be good to leave off swap to\n> reduce trashing).  There has to be a better way than a straight 128MB\n> default.\n>\n> The number which works on my 8GB desktop is going to kill the computer\n> in the trunk of my car with 48MB of ram.  I've actually seen a 700 MB\n> repository fail with `git-gc --aggressive` on a system with 4GB of ram\n> because it ran out of memory, it only worked by leaving off the\n> --aggressive option.\n\nIt seems to me that if \"git init\" creates a $GIT_DIR/info/attributes\nfile by default with a line like \"*.gz -delta\", then that will disable\nthe memory intensive deltra compression plumbing for those special\ncases where people need to track gzip archives (similarly another line\n\"*.bz2 -delta\" for bzip2 archives and so on). Since these files can't\nsupposedly be compressed much more, I think Git ought to have a\ndefault heuristic to not attempt any compression on them.\n\nCheers\n\nAlif\n"},{"id":"165176","messageId":"BANLkTi=nzY7wFTr3647SYkSMHx=AityRgw@mail.gmail.com","threadId":"26964","inReplyTo":"BANLkTik6XoWaehp7=5fxwcyhkTHWCO8-Eg@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-04-05T11:13:44Z","receivedAt":"2011-04-05T11:13:44Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 5, 2011 at 11:35 AM, Alif Wahid wrote:\n> It seems to me that if \"git init\" creates a $GIT_DIR/info/attributes\n> file by default with a line like \"*.gz -delta\", then that will disable\n> the memory intensive deltra compression plumbing for those special\n> cases where people need to track gzip archives (similarly another line\n> \"*.bz2 -delta\" for bzip2 archives and so on). Since these files can't\n> supposedly be compressed much more, I think Git ought to have a\n> default heuristic to not attempt any compression on them.\n\nI was thinking of very similar thing on my ride home. But I selected\nfiles on size, not extension. With the (hopefully coming soon)\nintroduction of pathspec magic specifier [1], we can teach git-attr to\nexpress \"files that have size in a range [a,b]\" (either a or b can be\ninfinite). The rest is like yours: applying -delta on selected files\nthen put such a rule with a default range in default template.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/169813/focus=169844\n-- \nDuy\n"},{"id":"165177","messageId":"BANLkTi=cBcsu1uehkLUP8-io9r1j-=jmNg@mail.gmail.com","threadId":"26964","inReplyTo":"BANLkTi=nzY7wFTr3647SYkSMHx=AityRgw@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Alif Wahid","fromEmail":"alif.wahid@gmail.com","sentAt":"2011-04-05T11:26:03Z","receivedAt":"2011-04-05T11:26:03Z","isPatch":false,"sender":{"key":"alif.wahid@gmail.com","avatar":null},"body":"On 5 April 2011 21:13, Nguyen Thai Ngoc Duy wrote:\n> On Tue, Apr 5, 2011 at 11:35 AM, Alif Wahid wrote:\n>> It seems to me that if \"git init\" creates a $GIT_DIR/info/attributes\n>> file by default with a line like \"*.gz -delta\", then that will disable\n>> the memory intensive deltra compression plumbing for those special\n>> cases where people need to track gzip archives (similarly another line\n>> \"*.bz2 -delta\" for bzip2 archives and so on). Since these files can't\n>> supposedly be compressed much more, I think Git ought to have a\n>> default heuristic to not attempt any compression on them.\n>\n> I was thinking of very similar thing on my ride home. But I selected\n> files on size, not extension. With the (hopefully coming soon)\n> introduction of pathspec magic specifier [1], we can teach git-attr to\n> express \"files that have size in a range [a,b]\" (either a or b can be\n> infinite). The rest is like yours: applying -delta on selected files\n> then put such a rule with a default range in default template.\n>\n> [1] http://thread.gmane.org/gmane.comp.version-control.git/169813/focus=169844\n\nYeah, makes sense.\n\nI also noticed the following thread regarding big file support. Most\nof the details there are related to this issue as well.\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/170649/focus=170649\n\nAlif\n"},{"id":"165186","messageId":"4D9B47D2.6050909@ira.uka.de","threadId":"26964","inReplyTo":"BANLkTinU7x16yp+y-HW9UhkVn9SftOJCcA@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Holger Hellmuth","fromEmail":"hellmuth@ira.uka.de","sentAt":"2011-04-05T16:48:18Z","receivedAt":"2011-04-05T16:48:18Z","isPatch":false,"sender":{"key":"hellmuth@ira.uka.de","avatar":null},"body":"On 04.04.2011 16:57, Nguyen Thai Ngoc Duy wrote:\n> On Mon, Apr 4, 2011 at 7:52 PM, Alif Wahid wrote:\n>> Hi Nicolas,\n>>\n>> On 4 April 2011 01:18, Nicolas Pitre wrote:\n>>>\n>>> Something you can try is to simply tell Git not to attempt any delta\n>>> compression on those tar files using gitattributes (see the man page of\n>>> the same name).\n>\n> Should we change the default to not delta if a blob exceeds predefined\n> limit (say 128M)? People who deliberately wants to delta them can\n> still set delta attr. 1.8.0 material maybe?\n\n\nIsn't this already done with the config variable core.bigFileThreshold ?\n\ndocumentation says: \"Files larger than this size are stored deflated, \nwithout attempting delta compression. ... Default is 512 MiB on all \nplatforms.\"\n"},{"id":"165187","messageId":"BANLkTikanSa3D1Bd8kSySPWQhcj1y8N+qA@mail.gmail.com","threadId":"26964","inReplyTo":"4D9B47D2.6050909@ira.uka.de","subject":"Re: Git exhausts memory.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-04-05T17:06:28Z","receivedAt":"2011-04-05T17:06:28Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, Apr 5, 2011 at 12:48, Holger Hellmuth wrote:\n> On 04.04.2011 16:57, Nguyen Thai Ngoc Duy wrote:\n>>\n>> Should we change the default to not delta if a blob exceeds predefined\n>> limit (say 128M)? People who deliberately wants to delta them can\n>> still set delta attr. 1.8.0 material maybe?\n>\n> Isn't this already done with the config variable core.bigFileThreshold ?\n>\n> documentation says: \"Files larger than this size are stored deflated,\n> without attempting delta compression. ... Default is 512 MiB on all\n> platforms.\"\n\nThis is only implemented inside of fast-import. pack-objects does not\nhonor this variable.\n\n-- \nShawn.\n"},{"id":"165192","messageId":"7vzko4mw44.fsf@alter.siamese.dyndns.org","threadId":"26964","inReplyTo":"BANLkTikanSa3D1Bd8kSySPWQhcj1y8N+qA@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-04-05T17:44:11Z","receivedAt":"2011-04-05T17:44:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce writes:\n\n> On Tue, Apr 5, 2011 at 12:48, Holger Hellmuth wrote:\n>> On 04.04.2011 16:57, Nguyen Thai Ngoc Duy wrote:\n>>>\n>>> Should we change the default to not delta if a blob exceeds predefined\n>>> limit (say 128M)? People who deliberately wants to delta them can\n>>> still set delta attr. 1.8.0 material maybe?\n>>\n>> Isn't this already done with the config variable core.bigFileThreshold ?\n>>\n>> documentation says: \"Files larger than this size are stored deflated,\n>> without attempting delta compression. ... Default is 512 MiB on all\n>> platforms.\"\n>\n> This is only implemented inside of fast-import. pack-objects does not\n> honor this variable.\n\nDo you mean perhaps we should?\n\n builtin/pack-objects.c | 8 ++++++--\n cache.h | 1 +\n config.c | 6 ++++++\n environment.c | 1 +\n fast-import.c | 5 -----\n 5 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b0503b2..f402a84 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1142,8 +1142,12 @@ static void get_object_details(void)\n \t\tsorted_by_offset[i] = objects + i;\n \tqsort(sorted_by_offset, nr_objects, sizeof(*sorted_by_offset), pack_offset_sort);\n \n-\tfor (i = 0; i < nr_objects; i++)\n-\t\tcheck_object(sorted_by_offset[i]);\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *entry = sorted_by_offset[i];\n+\t\tcheck_object(entry);\n+\t\tif (big_file_threshold <= entry->size)\n+\t\t\tentry->no_try_delta = 1;\n+\t}\n \n \tfree(sorted_by_offset);\n }\ndiff --git a/cache.h b/cache.h\nindex 2674f4c..316d85f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -573,6 +573,7 @@ extern int core_compression_seen;\n extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n+extern uintmax_t big_file_threshold;\n extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\ndiff --git a/config.c b/config.c\nindex 0abcada..d06fb19 100644\n--- a/config.c\n+++ b/config.c\n@@ -567,6 +567,12 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.bigfilethreshold\")) {\n+\t\tlong n = git_config_int(var, value);\n+\t\tbig_file_threshold = 0 < n ? n : 0;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.packedgitlimit\")) {\n \t\tpacked_git_limit = git_config_int(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex f4549d3..3d1ab51 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -35,6 +35,7 @@ int fsync_object_files;\n size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;\n size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;\n size_t delta_base_cache_limit = 16 * 1024 * 1024;\n+uintmax_t big_file_threshold = 512 * 1024 * 1024;\n const char *pager_program;\n int pager_use_color = 1;\n const char *editor_program;\ndiff --git a/fast-import.c b/fast-import.c\nindex 65d65bf..3e4e655 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -274,7 +274,6 @@ struct recent_command {\n /* Configured limits on output */\n static unsigned long max_depth = 10;\n static off_t max_packsize;\n-static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n static int force_update;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n@@ -3206,10 +3205,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tmax_packsize = git_config_ulong(k, v);\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"core.bigfilethreshold\")) {\n-\t\tlong n = git_config_int(k, v);\n-\t\tbig_file_threshold = 0 < n ? n : 0;\n-\t}\n \treturn git_default_config(k, v, cb);\n }\n \n"},{"id":"165209","messageId":"alpine.LFD.2.00.1104051655310.28032@xanadu.home","threadId":"26964","inReplyTo":"7vzko4mw44.fsf@alter.siamese.dyndns.org","subject":"Re: Git exhausts memory.","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-04-05T20:56:20Z","receivedAt":"2011-04-05T20:56:20Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 5 Apr 2011, Junio C Hamano wrote:\n\n> Shawn Pearce writes:\n> \n> > On Tue, Apr 5, 2011 at 12:48, Holger Hellmuth wrote:\n> >> On 04.04.2011 16:57, Nguyen Thai Ngoc Duy wrote:\n> >>>\n> >>> Should we change the default to not delta if a blob exceeds predefined\n> >>> limit (say 128M)? People who deliberately wants to delta them can\n> >>> still set delta attr. 1.8.0 material maybe?\n> >>\n> >> Isn't this already done with the config variable core.bigFileThreshold ?\n> >>\n> >> documentation says: \"Files larger than this size are stored deflated,\n> >> without attempting delta compression. ... Default is 512 MiB on all\n> >> platforms.\"\n> >\n> > This is only implemented inside of fast-import. pack-objects does not\n> > honor this variable.\n> \n> Do you mean perhaps we should?\n\nYes.\n\nAcked-by: Nicolas Pitre \n\n\n> builtin/pack-objects.c | 8 ++++++--\n> cache.h | 1 +\n> config.c | 6 ++++++\n> environment.c | 1 +\n> fast-import.c | 5 -----\n> 5 files changed, 14 insertions(+), 7 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index b0503b2..f402a84 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1142,8 +1142,12 @@ static void get_object_details(void)\n> \t\tsorted_by_offset[i] = objects + i;\n> \tqsort(sorted_by_offset, nr_objects, sizeof(*sorted_by_offset), pack_offset_sort);\n> \n> -\tfor (i = 0; i < nr_objects; i++)\n> -\t\tcheck_object(sorted_by_offset[i]);\n> +\tfor (i = 0; i < nr_objects; i++) {\n> +\t\tstruct object_entry *entry = sorted_by_offset[i];\n> +\t\tcheck_object(entry);\n> +\t\tif (big_file_threshold <= entry->size)\n> +\t\t\tentry->no_try_delta = 1;\n> +\t}\n> \n> \tfree(sorted_by_offset);\n> }\n> diff --git a/cache.h b/cache.h\n> index 2674f4c..316d85f 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -573,6 +573,7 @@ extern int core_compression_seen;\n> extern size_t packed_git_window_size;\n> extern size_t packed_git_limit;\n> extern size_t delta_base_cache_limit;\n> +extern uintmax_t big_file_threshold;\n> extern int read_replace_refs;\n> extern int fsync_object_files;\n> extern int core_preload_index;\n> diff --git a/config.c b/config.c\n> index 0abcada..d06fb19 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -567,6 +567,12 @@ static int git_default_core_config(const char *var, const char *value)\n> \t\treturn 0;\n> \t}\n> \n> +\tif (!strcmp(var, \"core.bigfilethreshold\")) {\n> +\t\tlong n = git_config_int(var, value);\n> +\t\tbig_file_threshold = 0 < n ? n : 0;\n> +\t\treturn 0;\n> +\t}\n> +\n> \tif (!strcmp(var, \"core.packedgitlimit\")) {\n> \t\tpacked_git_limit = git_config_int(var, value);\n> \t\treturn 0;\n> diff --git a/environment.c b/environment.c\n> index f4549d3..3d1ab51 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -35,6 +35,7 @@ int fsync_object_files;\n> size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;\n> size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;\n> size_t delta_base_cache_limit = 16 * 1024 * 1024;\n> +uintmax_t big_file_threshold = 512 * 1024 * 1024;\n> const char *pager_program;\n> int pager_use_color = 1;\n> const char *editor_program;\n> diff --git a/fast-import.c b/fast-import.c\n> index 65d65bf..3e4e655 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -274,7 +274,6 @@ struct recent_command {\n> /* Configured limits on output */\n> static unsigned long max_depth = 10;\n> static off_t max_packsize;\n> -static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n> static int force_update;\n> static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n> static int pack_compression_seen;\n> @@ -3206,10 +3205,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\tmax_packsize = git_config_ulong(k, v);\n> \t\treturn 0;\n> \t}\n> -\tif (!strcmp(k, \"core.bigfilethreshold\")) {\n> -\t\tlong n = git_config_int(k, v);\n> -\t\tbig_file_threshold = 0 < n ? n : 0;\n> -\t}\n> \treturn git_default_config(k, v, cb);\n> }\n> \n> \n"},{"id":"165223","messageId":"7vmxk4l4yb.fsf@alter.siamese.dyndns.org","threadId":"26964","inReplyTo":"alpine.LFD.2.00.1104051655310.28032@xanadu.home","subject":"Re: Git exhausts memory.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-04-05T22:16:12Z","receivedAt":"2011-04-05T22:16:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n>> > This is only implemented inside of fast-import. pack-objects does not\n>> > honor this variable.\n>> \n>> Do you mean perhaps we should?\n>\n> Yes.\n>\n> Acked-by: Nicolas Pitre \n\nI actually was somewhat unhappy to use uintmax_t type in the public header\nfor some reason I cannot quite explain (perhaps religious), and was hoping\nsomebody with more sanity than myself would stop me or show me a better\nway.\n\n>> diff --git a/cache.h b/cache.h\n>> index 2674f4c..316d85f 100644\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -573,6 +573,7 @@ extern int core_compression_seen;\n>> extern size_t packed_git_window_size;\n>> extern size_t packed_git_limit;\n>> extern size_t delta_base_cache_limit;\n>> +extern uintmax_t big_file_threshold;\n>> extern int read_replace_refs;\n>> extern int fsync_object_files;\n>> extern int core_preload_index;\n"},{"id":"165226","messageId":"BANLkTim-6L1ZeaPT3tVWmyhj-rX1iURQ3g@mail.gmail.com","threadId":"26964","inReplyTo":"7vmxk4l4yb.fsf@alter.siamese.dyndns.org","subject":"Re: Git exhausts memory.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-04-05T22:19:30Z","receivedAt":"2011-04-05T22:19:30Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, Apr 5, 2011 at 18:16, Junio C Hamano wrote:\n> Nicolas Pitre writes:\n>\n>>> > This is only implemented inside of fast-import. pack-objects does not\n>>> > honor this variable.\n>>>\n>>> Do you mean perhaps we should?\n>>\n>> Yes.\n>>\n>> Acked-by: Nicolas Pitre \n>\n> I actually was somewhat unhappy to use uintmax_t type in the public header\n> for some reason I cannot quite explain (perhaps religious), and was hoping\n> somebody with more sanity than myself would stop me or show me a better\n> way.\n\nunsigned long? Without even looking at the source, I bet that is the\ntype used by pack-objects for the size member that you are comparing\nagainst.\n\n-- \nShawn.\n"},{"id":"165242","messageId":"alpine.LFD.2.00.1104052032290.28032@xanadu.home","threadId":"26964","inReplyTo":"7vmxk4l4yb.fsf@alter.siamese.dyndns.org","subject":"Re: Git exhausts memory.","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-04-06T00:34:17Z","receivedAt":"2011-04-06T00:34:17Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 5 Apr 2011, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> >> > This is only implemented inside of fast-import. pack-objects does not\n> >> > honor this variable.\n> >> \n> >> Do you mean perhaps we should?\n> >\n> > Yes.\n> >\n> > Acked-by: Nicolas Pitre \n> \n> I actually was somewhat unhappy to use uintmax_t type in the public header\n> for some reason I cannot quite explain (perhaps religious), and was hoping\n> somebody with more sanity than myself would stop me or show me a better\n> way.\n\nJust use unsigned long. Everywhere we have object size, it is stored as \nunsigned long.\n\n\nNicolas\n"},{"id":"165284","messageId":"BANLkTinOzkHMpr5Yg7dixMyTSHBXCsO7Qw@mail.gmail.com","threadId":"26964","inReplyTo":"7vzko4mw44.fsf@alter.siamese.dyndns.org","subject":"Re: Git exhausts memory.","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2011-04-06T15:51:26Z","receivedAt":"2011-04-06T15:51:26Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Tue, Apr 5, 2011 at 1:44 PM, Junio C Hamano wrote:\n>  builtin/pack-objects.c |    8 ++++++--\n>  cache.h                |    1 +\n>  config.c               |    6 ++++++\n>  environment.c          |    1 +\n>  fast-import.c          |    5 -----\n>  5 files changed, 14 insertions(+), 7 deletions(-)\n\nThis will be white-spaced damaged by Gmail, but anyway:\n\ndiff --git i/Documentation/config.txt w/Documentation/config.txt\nindex 750c86d..91aa9be 100644\n--- i/Documentation/config.txt\n+++ w/Documentation/config.txt\n@@ -443,7 +443,6 @@ be delta compressed, but larger binary media files won't be.\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n +\n-Currently only linkgit:git-fast-import[1] honors this setting.\n\n core.excludesfile::\n \tIn addition to '.gitignore' (per-directory) and\n\n\nj.\n"},{"id":"165286","messageId":"7vtyebjq58.fsf@alter.siamese.dyndns.org","threadId":"26964","inReplyTo":"BANLkTinOzkHMpr5Yg7dixMyTSHBXCsO7Qw@mail.gmail.com","subject":"Re: Git exhausts memory.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-04-06T16:33:39Z","receivedAt":"2011-04-06T16:33:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jay Soffian writes:\n\n> This will be white-spaced damaged by Gmail, but anyway:\n\nI've already done that last night. Thanks.\n"}]}