{"thread":{"id":"30793","subject":"Git Garbage Collect Error.","startedAt":"2012-06-13T10:27:04Z","lastAt":"2012-07-14T03:36:30Z","messageCount":4,"participants":["Thomas Lucas","Jeff King","Philippe Vaucher","sascha-ml@babbelbox.org"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"193536","messageId":"4FD86AF8.1050100@zuken.co.uk","threadId":"30793","inReplyTo":null,"subject":"Git Garbage Collect Error.","fromName":"Thomas Lucas","fromEmail":"toml@zuken.co.uk","sentAt":"2012-06-13T10:27:04Z","receivedAt":"2012-06-13T10:27:04Z","isPatch":false,"sender":{"key":"toml@zuken.co.uk","avatar":null},"body":"\nHi,\n\nHopefully this is the right place to send bug reports... The community page \n\"http://git-scm.com/community\" suggests that it is.\n\n\n      Introduction\n\nI am creating a large GIT repository fetching from a large SVN repository, as an \nexperiment initially. I usually use GIT repositories interfacing to parts of the \nSVN repository.\n\n\n      Defect\n\nDuring garbage collection (git gc) it encountered the following error:\n\ngit gc | git gc --prune :\n\n    Counting objects: 856758, done.\n    Delta compression using up to 2 threads.\n    fatal: Out of memory, malloc failed (tried to allocate 303237121 bytes)\n    error: failed to run repack\n\ngit gc --aggressive:\n\n    Counting objects: 856758, done.\n    Delta compression using up to 2 threads.\n    fatal: Out of memory, malloc failed (tried to allocate 291942401 bytes)\n    error: failed to run repack\n\nAt the moment the bare repository is about 4Gb in size and about 2/3rds the way \nthrough fetching.\n\nThe compression gets over 90% of the way through before this error occurs, but I \ndon't think any compression results are kept, because when you repeat it has the \nsame amount of work to do.\n\nInitially this happen during an automatic gc during the fetch process. This \naborted the fetch.\n\nMy system is XP64 2 core with 4Gb of memory and plenty of virtual memory.\n\n\n      Comments\n\nIf this a genuine limitation due to the size of an object and memory handling \nlimitations, then perhaps the error could be caught and the successful results \nkept. Ie. do a partial compression. That way the process could continue.\n\n\n      Background\n\nMy requirement is to have GIT repositories of a source directory with all SVN \nbranches included so that I can more easily merge and compare branches using \nGIT. However for even small source directories it takes weeks to fetch from the \nSVN respository (including all tags and branches), whereas fetching just the \ntrunk takes a few hours. The SVN repository has over 90000 revisions. I am aware \nthat I can fetch a sub-set of revisions (I don't want to at the moment), but \nI've found no way to fetch a sub-set of branches.\n\nMy config is as follows:\n\n[svn-remote \"svn\"]\n         url = svn://svn\n         fetch = trunk:refs/remotes/svn/trunk\n         branches = branches/*:refs/remotes/svn/*\n         tags = tags/*:refs/remotes/svn/tags/*\n\nI set this up using:\n\ngit svn init --prefix=svn/ --stdlayout --no-minimize-url svn://svn\n\nTo do this for individual directories I have to do the following:\n\ngit svn init --prefix=svn/ --stdlayout --no-minimize-url \nsvn://svn/trunk/source/<dir>\n\nand then edit the config manually so that:\n\n[svn-remote \"svn\"]\n         url = svn://svn\n         fetch = trunk/source/<dir>:refs/remotes/svn/trunk\n         branches = branches/*/source/<dir>:refs/remotes/svn/*\n         tags = tags/*/source/<dir>:refs/remotes/svn/tags/*\n\nThis works ok but I couldn't get this result by using \"git svn init\" directly. \nMaybe I've missed something.\n\nRegards,\nTom.\n"},{"id":"194957","messageId":"20120712093221.GA4443@sigill.intra.peff.net","threadId":"30793","inReplyTo":"4FD86AF8.1050100@zuken.co.uk","subject":"Re: Git Garbage Collect Error.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-07-12T09:32:21Z","receivedAt":"2012-07-12T09:32:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 13, 2012 at 11:27:04AM +0100, Thomas Lucas wrote:\n\n> Hopefully this is the right place to send bug reports... The\n> community page \"http://git-scm.com/community\" suggests that it is.\n\nIt is the right place. Sorry that you did not get any response before\nnow.\n\n> During garbage collection (git gc) it encountered the following error:\n> \n> git gc | git gc --prune :\n> \n>    Counting objects: 856758, done.\n>    Delta compression using up to 2 threads.\n>    fatal: Out of memory, malloc failed (tried to allocate 303237121 bytes)\n>    error: failed to run repack\n\nPacking can be memory hungry if you have a lot of large objects (we may\nhold several large objects in memory while comparing them for deltas).\nIt is also worse with 2 threads, as they will be working simultaneously,\nbut in the same memory space.\n\n> The compression gets over 90% of the way through before this error\n> occurs, but I don't think any compression results are kept, because\n> when you repeat it has the same amount of work to do.\n\nRight. Nothing is written during compression; we are just coming up with\na list of deltas to perform during the writing phase.\n\n> My system is XP64 2 core with 4Gb of memory and plenty of virtual memory.\n\nUnfortunately, I believe that the msysgit build is 32-bit, which means\nyou are probably not even getting to use all 4Gb of your address space\n(my impression is that without special flags, 32-bit Windows processes\nare limited to 2Gb of address space).\n\nI'd first try doing the pack single-threaded by setting the pack.threads\nconfig option to 1. If that doesn't work, you might try setting\npack.windowMemory to limit the delta search based on available memory\n(usually it is limited by number of objects). If the large blobs are\nones that do not delta well anyway (e.g., compressed media files), you\nmight also consider setting the \"-delta\" attribute for them to skip\ndelta compression entirely.\n\n-Peff\n"},{"id":"194962","messageId":"CAGK7Mr5aAkQOu_LxvW7q13dup5GYpfBQUcUEeWsyYM+9cTYX=g@mail.gmail.com","threadId":"30793","inReplyTo":"4FD86AF8.1050100@zuken.co.uk","subject":"Re: Git Garbage Collect Error.","fromName":"Philippe Vaucher","fromEmail":"philippe.vaucher@gmail.com","sentAt":"2012-07-12T12:34:30Z","receivedAt":"2012-07-12T12:34:30Z","isPatch":false,"sender":{"key":"philippe.vaucher@gmail.com","avatar":null},"body":"> At the moment the bare repository is about 4Gb in size and about 2/3rds the way through fetching.\n\nThat's a big repo. Lots of binary files in it?\nDoes git fsck run normally? Does it report a lot of dangling blogs/commits/etc?\n\nPhilippe\n"},{"id":"195036","messageId":"7478187.7CcD6s6PIO@toshi","threadId":"30793","inReplyTo":"20120712093221.GA4443@sigill.intra.peff.net","subject":"Re: Git Garbage Collect Error.","fromName":"","fromEmail":"sascha-ml@babbelbox.org","sentAt":"2012-07-14T03:36:30Z","receivedAt":"2012-07-14T03:36:30Z","isPatch":false,"sender":{"key":"sascha-ml@babbelbox.org","avatar":null},"body":"On Thursday 12 July 2012 05:32:21 Jeff King wrote:\n\n> [...] which means you are probably not even getting to use all 4Gb of your\n> address space (my impression is that without special flags, 32-bit Windows\n> processes are limited to 2Gb of address space).\n\nIndeed, that's how windows partitions memory on 32-Bit Systems. See:\n\nhttp://msdn.microsoft.com/en-us/library/windows/desktop/aa366912.aspx\n\nAs it's always with that strange company, they don't spend a word about how \nthey do it in the emulated 32 bit environment. However, a short testing \nreveals that a 32 bit process running on 64 bit Windows 7 with 6 GiB memory is \nnot able to malloc() more than 1 GiB at once (which is not a big suprise at \nall - as malloc'ed memory has to be continuous inside the address space).\n\nSo one might guess that there is no difference in memory partitioning for 32 \nbit processes running on 64 bit OS.\n\nSaCu\n"}]}