{"thread":{"id":"27527","subject":"out of memory error with git push and pull","startedAt":"2011-06-01T22:33:33Z","lastAt":"2011-06-02T22:13:14Z","messageCount":3,"participants":["Qingning Huo","Joey Hess"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"169159","messageId":"BANLkTin3-XnVsVd1-CAiWDBzQG6m=a4Rvw@mail.gmail.com","threadId":"27527","inReplyTo":null,"subject":"out of memory error with git push and pull","fromName":"Qingning Huo","fromEmail":"qingninghuo@gmail.com","sentAt":"2011-06-01T22:33:33Z","receivedAt":"2011-06-01T22:33:33Z","isPatch":false,"sender":{"key":"qingninghuo@gmail.com","avatar":null},"body":"Hi All,\n\nI tried to use git to manage my digital photos but encountered some\nproblems. The typical file sizes are hundreds of KB or a few MB. In\ntotal, about 15GB data in about 10,000 files. My intention is to get\nthem into a git repository and cloned into a few computers. Probably I\nwill make some occasionally changes like editing and deleting. But I\nthink most of the files would stay at version one.\n\nThe setup I used is a centralized repository for pulling from and\npushing into. There is a gitweb interface for the centralised repo.\n\nI started by creating a small repository on the server (ubuntu), and\nkeeps pushing data into it from a windows machine (using cygwin). Half\nway through the process, (after pushed about 8GB of data), I found\nthat I cannot run git push any more. This is the error message I got:\n\n$ git push\nCounting objects: 621, done.\nDelta compression using up to 4 threads.\nfatal: Out of memory? mmap failed: Cannot allocate memory\nerror: pack-objects died with strange error\nerror: failed to push some refs to 'ssh://huo@ubuntu/mnt/share/git/photo.git'\n\nAt the same time, I found that I cannot pull from this repository either.\n\n$ git pull\nremote: Counting objects: 8088, done.\nerror: pack-objects died of signal 983/8057)\nerror: git upload-pack: git-pack-objects died with error.\nfatal: git upload-pack: aborting due to possible repository corruption\non the remote side.\nremote: aborting due to possible repository corruption on the remote side.\nfatal: protocol error: bad pack header\n\n[The second line is probably \"died of signal 9\", because it was\ncounting ???/8057 upwards before the crash.]\n\nI wonder whether anyone has tried using git in a similar scenario. Is\ngit capable of handling this kind of data? And, are there any settings\nand/or command line options that I should use? I had a quick look of\ngit help push (and pull/fetch) but cannot see anything obvious.\n\nBTW, I am using git version 1.7.0.4 on the ubuntu server, and version\n1.7.2.3 for cygwin on the client side.\n\nThanks in advance.\n\nQingning\n"},{"id":"169173","messageId":"20110602044603.GB5081@gnu.kitenet.net","threadId":"27527","inReplyTo":"BANLkTin3-XnVsVd1-CAiWDBzQG6m=a4Rvw@mail.gmail.com","subject":"Re: out of memory error with git push and pull","fromName":"Joey Hess","fromEmail":"joey@kitenet.net","sentAt":"2011-06-02T04:46:03Z","receivedAt":"2011-06-02T04:46:03Z","isPatch":false,"sender":{"key":"joey@kitenet.net","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Qingning Huo wrote:\n> I tried to use git to manage my digital photos but encountered some\n> problems. The typical file sizes are hundreds of KB or a few MB. In\n> total, about 15GB data in about 10,000 files. My intention is to get\n> them into a git repository and cloned into a few computers. Probably I\n> will make some occasionally changes like editing and deleting. But I\n> think most of the files would stay at version one.\n\nI try not to mention git-annex too much here, but this is a perfect\nuse-case for it. http://git-annex.branchable.com/ \n\nWell, it would be more perfect if you had enough data in your repo that\nyou didn't necessarily want to clone it all to every computer. Like so:\n\n# git annex status\nlocal annex size: 58 megabytes\ntotal annex keys: 38158\ntotal annex size: 6 terabytes\n\n:)\n\n> I wonder whether anyone has tried using git in a similar scenario. Is\n> git capable of handling this kind of data? And, are there any settings\n> and/or command line options that I should use? I had a quick look of\n> git help push (and pull/fetch) but cannot see anything obvious.\n\nThere is a tunable you can use to improve things, see core.bigFileThreshold\n\nThat originally came from this project.\nhttp://caca.zoy.org/wiki/git-bigfiles -- it may have some other\nimprovements that have not landed in git, I'm not sure.\n\n-- \nsee shy jo\n"},{"id":"169225","messageId":"BANLkTimSoqbg6LgunovaKRefX78OH=xLMg@mail.gmail.com","threadId":"27527","inReplyTo":"20110602044603.GB5081@gnu.kitenet.net","subject":"Re: out of memory error with git push and pull","fromName":"Qingning Huo","fromEmail":"qingninghuo@gmail.com","sentAt":"2011-06-02T22:13:14Z","receivedAt":"2011-06-02T22:13:14Z","isPatch":false,"sender":{"key":"qingninghuo@gmail.com","avatar":null},"body":"Hi Joey,\n\nOn Thu, Jun 2, 2011 at 5:46 AM, Joey Hess <joey@kitenet.net> wrote:\n> Qingning Huo wrote:\n>> I tried to use git to manage my digital photos but encountered some\n>> problems. The typical file sizes are hundreds of KB or a few MB. In\n>> total, about 15GB data in about 10,000 files. My intention is to get\n>> them into a git repository and cloned into a few computers. Probably I\n>> will make some occasionally changes like editing and deleting. But I\n>> think most of the files would stay at version one.\n>\n> I try not to mention git-annex too much here, but this is a perfect\n> use-case for it. http://git-annex.branchable.com/\n>\n> Well, it would be more perfect if you had enough data in your repo that\n> you didn't necessarily want to clone it all to every computer. Like so:\n>\n> # git annex status\n> local annex size: 58 megabytes\n> total annex keys: 38158\n> total annex size: 6 terabytes\n>\n> :)\n\nThanks a lot for the pointer. I'd love to use git-annex if I can get\nmy hand to it.\nI had a look of the web site and searched a bit on the web, but there does not\nseem to be an easy way to install it on windows/cygwin.\n\nI might try the bigFileThreshold setting first. And maybe git-bigfiles.\n\n>\n>> I wonder whether anyone has tried using git in a similar scenario. Is\n>> git capable of handling this kind of data? And, are there any settings\n>> and/or command line options that I should use? I had a quick look of\n>> git help push (and pull/fetch) but cannot see anything obvious.\n>\n> There is a tunable you can use to improve things, see core.bigFileThreshold\n>\n> That originally came from this project.\n> http://caca.zoy.org/wiki/git-bigfiles -- it may have some other\n> improvements that have not landed in git, I'm not sure.\n>\n> --\n> see shy jo\n>\n\nThanks\nQingning\n"}]}