{"thread":{"id":"16292","subject":"running out of memory when doing a clone of a large repository?","startedAt":"2008-11-12T18:49:52Z","lastAt":"2008-11-13T20:09:00Z","messageCount":3,"participants":["Paul E. Rybski","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"95614","messageId":"491B2550.1050005@cs.cmu.edu","threadId":"16292","inReplyTo":null,"subject":"running out of memory when doing a clone of a large repository?","fromName":"Paul E. Rybski","fromEmail":"prybski@cs.cmu.edu","sentAt":"2008-11-12T18:49:52Z","receivedAt":"2008-11-12T18:49:52Z","isPatch":false,"sender":{"key":"prybski@cs.cmu.edu","avatar":null},"body":"Hi,\n\tI've recently started evaluating git for use on one of my projects.  I\nused git-svn to convert a fairly large subversion repository and one of\nits branches into two separate git repositories following the\ninstructions found here:\n\nhttp://www.simplisticcomplexity.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/\n\n>From that, I created a 1.5G repository from the trunk of the subversion\nrepo and a 1.1G repository from one of the branches.  I then tried to\nclone the repositories from one machine to another via ssh and the\nsmaller 1.1G repository of the subversion branch (with 1932 commits)\ncame over just fine but the larger 1.5G repository of the subversion\ntrunk (5865 commits) died with the following error:\n\nremote: Counting objects: 48415, done.\nremote: warning: suboptimal pack - out of memory\nerror: git-upload-pack: git-pack-objects died with error.\nfatal: git-upload-pack: aborting due to possible repository corruption\non the remote side.\nremote: aborting due to possible repository corruption on the remote side.\nfatal: early EOF\nfatal: index-pack failed\n\nIs this simply because the git repository is too large for the machine\nthat's trying to send it over ssh?  Is there a way to restrict the\nmemory usage of git or do I need to get more RAM or somehow not import\nthe full subversion history of my original trunk?\n\nI'm using Ubuntu 8.04.1 on both machines and using the Ubuntu packages\nof git version 1.5.4.3.  The machine holding the repository is a 5-year\nold AthlonXP 3200 with 1G RAM and 32-bit Ubuntu.  The machine I'm\ncloning the repository on is a more recent Intel Dual Core Quad Q6600\nwith 4GRAM and 64-bit Ubuntu.\n\nAny advice would be greatly appreciated.\n\nThanks!\n\n-Paul\n\n-- \nPaul E. Rybski, Ph.D., Systems Scientist\nThe Robotics Institute, Carnegie Mellon University\nPhone: 412-268-7417, Fax: 412-268-7350\nWeb: http://www.cs.cmu.edu/~prybski\n"},{"id":"95661","messageId":"alpine.LFD.2.00.0811122206550.27509@xanadu.home","threadId":"16292","inReplyTo":"491B2550.1050005@cs.cmu.edu","subject":"Re: running out of memory when doing a clone of a large repository?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-11-13T03:22:35Z","receivedAt":"2008-11-13T03:22:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 12 Nov 2008, Paul E. Rybski wrote:\n\n> Hi,\n> \tI've recently started evaluating git for use on one of my projects.  I\n> used git-svn to convert a fairly large subversion repository and one of\n> its branches into two separate git repositories following the\n> instructions found here:\n> \n> http://www.simplisticcomplexity.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/\n> \n> From that, I created a 1.5G repository from the trunk of the subversion\n> repo and a 1.1G repository from one of the branches.  I then tried to\n> clone the repositories from one machine to another via ssh and the\n> smaller 1.1G repository of the subversion branch (with 1932 commits)\n> came over just fine but the larger 1.5G repository of the subversion\n> trunk (5865 commits) died with the following error:\n> \n> remote: Counting objects: 48415, done.\n> remote: warning: suboptimal pack - out of memory\n> error: git-upload-pack: git-pack-objects died with error.\n> fatal: git-upload-pack: aborting due to possible repository corruption\n> on the remote side.\n> remote: aborting due to possible repository corruption on the remote side.\n> fatal: early EOF\n> fatal: index-pack failed\n> \n> Is this simply because the git repository is too large for the machine\n> that's trying to send it over ssh?\n\nPossibly.\n\n> Is there a way to restrict the memory usage of git or do I need to get \n> more RAM or somehow not import the full subversion history of my \n> original trunk?\n\nIt is possible that the inport from svn didn't produce a fully packed \nrepository.  Try a \"git repack -a -f -d\" on the server machine. If you \nget the same out-of-memory problem then ideally you should copy the \nwhole directory to a bigger machine and run 'git repack -a -f -d\" there.  \nThen you may copy it back to the small machine and subsequent clone \nrequests should be just fine.\n\nYou could also investigate the pack.windowMemory configuration variable \non the server side.  If you have lots of big files then setting this to \n64m for example may help.\n\n> I'm using Ubuntu 8.04.1 on both machines and using the Ubuntu packages\n> of git version 1.5.4.3.\n\nThere was a bug in versions prior to v1.5.6.4 affecting memory usage.  \nUpgrading to this version or later on the server machine might help too.\n\n> The machine holding the repository is a 5-year\n> old AthlonXP 3200 with 1G RAM and 32-bit Ubuntu.  The machine I'm\n> cloning the repository on is a more recent Intel Dual Core Quad Q6600\n> with 4GRAM and 64-bit Ubuntu.\n> \n> Any advice would be greatly appreciated.\n> \n> Thanks!\n> \n> -Paul\n> \n> -- \n> Paul E. Rybski, Ph.D., Systems Scientist\n> The Robotics Institute, Carnegie Mellon University\n> Phone: 412-268-7417, Fax: 412-268-7350\n> Web: http://www.cs.cmu.edu/~prybski\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n\n\nNicolas\n"},{"id":"95718","messageId":"491C895C.5010409@cs.cmu.edu","threadId":"16292","inReplyTo":"alpine.LFD.2.00.0811122206550.27509@xanadu.home","subject":"Re: running out of memory when doing a clone of a large repository?","fromName":"Paul E. Rybski","fromEmail":"prybski@cs.cmu.edu","sentAt":"2008-11-13T20:09:00Z","receivedAt":"2008-11-13T20:09:00Z","isPatch":false,"sender":{"key":"prybski@cs.cmu.edu","avatar":null},"body":"Hi,\n\tThanks for the great suggestions.  It turns out that running the command:\n\ngit repack -a -f -d\n\non the newly converted git repository seemed to fix my immediate\ncrashing-while-cloning problem.  I was able to clone via ssh from the\nolder Ubuntu machine with 1G of RAM using git version 1.5.4.3 (the\npackaged version in Hardy Heron).\n\nI also downloaded what seems to be the latest of the 1.5 series,\n1.5.6.5, installed that from source, and successfully tested that out.\nI'll try using that to avoid some of the memory problems of the previous\nversions.  As I'm still comparatively a complete newbie when it comes to\ngit, I don't have a clue yet as to what's in the 1.6 codebase but\nhopefully I'll be able to determine which version I want to work with soon.\n\nThanks again for the help!  I really appreciate it.\n\n-Paul\n\nNicolas Pitre wrote:\n> On Wed, 12 Nov 2008, Paul E. Rybski wrote:\n> \n>> Hi,\n>> \tI've recently started evaluating git for use on one of my projects.  I\n>> used git-svn to convert a fairly large subversion repository and one of\n>> its branches into two separate git repositories following the\n>> instructions found here:\n>>\n>> http://www.simplisticcomplexity.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/\n>>\n>> From that, I created a 1.5G repository from the trunk of the subversion\n>> repo and a 1.1G repository from one of the branches.  I then tried to\n>> clone the repositories from one machine to another via ssh and the\n>> smaller 1.1G repository of the subversion branch (with 1932 commits)\n>> came over just fine but the larger 1.5G repository of the subversion\n>> trunk (5865 commits) died with the following error:\n>>\n>> remote: Counting objects: 48415, done.\n>> remote: warning: suboptimal pack - out of memory\n>> error: git-upload-pack: git-pack-objects died with error.\n>> fatal: git-upload-pack: aborting due to possible repository corruption\n>> on the remote side.\n>> remote: aborting due to possible repository corruption on the remote side.\n>> fatal: early EOF\n>> fatal: index-pack failed\n>>\n>> Is this simply because the git repository is too large for the machine\n>> that's trying to send it over ssh?\n> \n> Possibly.\n> \n>> Is there a way to restrict the memory usage of git or do I need to get \n>> more RAM or somehow not import the full subversion history of my \n>> original trunk?\n> \n> It is possible that the inport from svn didn't produce a fully packed \n> repository.  Try a \"git repack -a -f -d\" on the server machine. If you \n> get the same out-of-memory problem then ideally you should copy the \n> whole directory to a bigger machine and run 'git repack -a -f -d\" there.  \n> Then you may copy it back to the small machine and subsequent clone \n> requests should be just fine.\n> \n> You could also investigate the pack.windowMemory configuration variable \n> on the server side.  If you have lots of big files then setting this to \n> 64m for example may help.\n> \n>> I'm using Ubuntu 8.04.1 on both machines and using the Ubuntu packages\n>> of git version 1.5.4.3.\n> \n> There was a bug in versions prior to v1.5.6.4 affecting memory usage.  \n> Upgrading to this version or later on the server machine might help too.\n> \n>> The machine holding the repository is a 5-year\n>> old AthlonXP 3200 with 1G RAM and 32-bit Ubuntu.  The machine I'm\n>> cloning the repository on is a more recent Intel Dual Core Quad Q6600\n>> with 4GRAM and 64-bit Ubuntu.\n>>\n>> Any advice would be greatly appreciated.\n>>\n>> Thanks!\n>>\n>> -Paul\n"}]}