{"thread":{"id":"30405","subject":"Large repo and pack.packsizelimit","startedAt":"2012-05-03T11:57:58Z","lastAt":"2012-05-10T11:42:09Z","messageCount":10,"participants":["th.acker66@arcor.de","Jeff King","Nicolas Pitre","Thomas","Nguyen Thai Ngoc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"190635","messageId":"37267143.413194.1336046278583.JavaMail.ngmail@webmail07.arcor-online.net","threadId":"30405","inReplyTo":null,"subject":"Large repo and pack.packsizelimit","fromName":"","fromEmail":"th.acker66@arcor.de","sentAt":"2012-05-03T11:57:58Z","receivedAt":"2012-05-03T11:57:58Z","isPatch":false,"sender":{"key":"th.acker@arcor.de","avatar":"https://avatars.githubusercontent.com/u/1358536?v=4"},"body":"Hello,\n\nI am using MSysgit 1.7.9 on WinXP 32bit and have a very large repo (10GB in .git; 20GB in source tree).\nI had to set pack.packsizelimit=1024MB to prevent \"out of memory\" during repacking in git-gc \nand everything seemed to work fine.\n\nWhen I tried to clone this repo an \"out of memory\" occured because the packs to be transferred\nby the git protocol are not limited by pack.packsizelimit. I \"fixed\" this by setting transfer.unpackLimit=100000\nand thus transferring only loose objects. This is very slow but it works.\n\nIn this cloned repo now git-gc again causes \"out of memory\" because it tries to pack all loose\nobjects in one go thereby seemingly not respecting pack.packsizelimit ... \n(Setting --window-memory=512m in git-repack did not help here.)\n\nAm I doing anything wrong here or is this a bug/feature in git?\n\nBTW1 Repo is very large but contains only one really large file with 1.2GB; all other files are smaller than 256MB.\nBTW2 I cannot use 1.7.10 due to the http authorization bug.\n\n\n---\nThomas\n"},{"id":"191150","messageId":"20120508203137.GA15707@sigill.intra.peff.net","threadId":"30405","inReplyTo":"37267143.413194.1336046278583.JavaMail.ngmail@webmail07.arcor-online.net","subject":"Re: Large repo and pack.packsizelimit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-08T20:31:37Z","receivedAt":"2012-05-08T20:31:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 03, 2012 at 01:57:58PM +0200, th.acker66@arcor.de wrote:\n\n> I am using MSysgit 1.7.9 on WinXP 32bit and have a very large repo\n> (10GB in .git; 20GB in source tree). I had to set\n> pack.packsizelimit=1024MB to prevent \"out of memory\" during repacking\n> in git-gc and everything seemed to work fine.\n> \n> When I tried to clone this repo an \"out of memory\" occured because the\n> packs to be transferred by the git protocol are not limited by\n> pack.packsizelimit.\n\nYes, pack-objects respects pack.packsizelimit when creating local packs,\nbut incoming packs from the network (which are processed by index-pack)\nare not split.\n\nThis should be fixed in git. Unfortunately, I don't know that it is as\ntrivial as just splitting the incoming stream; we would also have to\nmake sure that there were no cross-pack deltas in the result.\n\n-Peff\n"},{"id":"191151","messageId":"alpine.LFD.2.02.1205081709010.21030@xanadu.home","threadId":"30405","inReplyTo":"20120508203137.GA15707@sigill.intra.peff.net","subject":"Re: Large repo and pack.packsizelimit","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2012-05-08T21:13:13Z","receivedAt":"2012-05-08T21:13:13Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2012, Jeff King wrote:\n\n> On Thu, May 03, 2012 at 01:57:58PM +0200, th.acker66@arcor.de wrote:\n> \n> > I am using MSysgit 1.7.9 on WinXP 32bit and have a very large repo\n> > (10GB in .git; 20GB in source tree). I had to set\n> > pack.packsizelimit=1024MB to prevent \"out of memory\" during repacking\n> > in git-gc and everything seemed to work fine.\n> > \n> > When I tried to clone this repo an \"out of memory\" occured because the\n> > packs to be transferred by the git protocol are not limited by\n> > pack.packsizelimit.\n> \n> Yes, pack-objects respects pack.packsizelimit when creating local packs,\n> but incoming packs from the network (which are processed by index-pack)\n> are not split.\n> \n> This should be fixed in git. Unfortunately, I don't know that it is as\n> trivial as just splitting the incoming stream; we would also have to\n> make sure that there were no cross-pack deltas in the result.\n\nIMHO this is the wrong fix. The pack size limit was created to deal \nwith storage media with limited capacity. In this case, the repack \nprocess should be told to limit its memory usage, and pack-index should \nsimply be taught to cope.\n\n\nNicolas\n"},{"id":"191152","messageId":"20120508212012.GA20044@sigill.intra.peff.net","threadId":"30405","inReplyTo":"alpine.LFD.2.02.1205081709010.21030@xanadu.home","subject":"Re: Large repo and pack.packsizelimit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-08T21:20:12Z","receivedAt":"2012-05-08T21:20:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 08, 2012 at 05:13:13PM -0400, Nicolas Pitre wrote:\n\n> > This should be fixed in git. Unfortunately, I don't know that it is as\n> > trivial as just splitting the incoming stream; we would also have to\n> > make sure that there were no cross-pack deltas in the result.\n> \n> IMHO this is the wrong fix. The pack size limit was created to deal \n> with storage media with limited capacity. In this case, the repack \n> process should be told to limit its memory usage, and pack-index should \n> simply be taught to cope.\n\nHmm, you're right. I was thinking it helped to deal with memory\naddressing issues for 32-bit systems, but I guess\ncore.packedGitWindowSize should be handling that. IOW, the 10G packfile\nshould work just fine for normal access.\n\nHowever, the OP did say he got an \"out of memory\" error during the\nclone. So maybe there is a problem to be fixed in index-pack there.\n\n-Peff\n"},{"id":"191154","messageId":"alpine.LFD.2.02.1205081751011.21030@xanadu.home","threadId":"30405","inReplyTo":"20120508212012.GA20044@sigill.intra.peff.net","subject":"Re: Large repo and pack.packsizelimit","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2012-05-08T21:52:37Z","receivedAt":"2012-05-08T21:52:37Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2012, Jeff King wrote:\n\n> On Tue, May 08, 2012 at 05:13:13PM -0400, Nicolas Pitre wrote:\n> \n> > > This should be fixed in git. Unfortunately, I don't know that it is as\n> > > trivial as just splitting the incoming stream; we would also have to\n> > > make sure that there were no cross-pack deltas in the result.\n> > \n> > IMHO this is the wrong fix. The pack size limit was created to deal \n> > with storage media with limited capacity. In this case, the repack \n> > process should be told to limit its memory usage, and pack-index should \n> > simply be taught to cope.\n> \n> Hmm, you're right. I was thinking it helped to deal with memory\n> addressing issues for 32-bit systems, but I guess\n> core.packedGitWindowSize should be handling that. IOW, the 10G packfile\n> should work just fine for normal access.\n> \n> However, the OP did say he got an \"out of memory\" error during the\n> clone. So maybe there is a problem to be fixed in index-pack there.\n\nWas the OOM on the remote side (pack-objects) or on the local side \n(index-pack) ?\n\n\nNicolas\n"},{"id":"191189","messageId":"loom.20120509T113505-740@post.gmane.org","threadId":"30405","inReplyTo":"alpine.LFD.2.02.1205081751011.21030@xanadu.home","subject":"Re: Large repo and pack.packsizelimit","fromName":"Thomas","fromEmail":"th.acker66@arcor.de","sentAt":"2012-05-09T09:36:10Z","receivedAt":"2012-05-09T09:36:10Z","isPatch":false,"sender":{"key":"th.acker@arcor.de","avatar":"https://avatars.githubusercontent.com/u/1358536?v=4"},"body":"Nicolas Pitre fluxnic.net> writes:\n\n> \n> On Tue, 8 May 2012, Jeff King wrote:\n> \n> > On Tue, May 08, 2012 at 05:13:13PM -0400, Nicolas Pitre wrote:\n> > \n> > > > This should be fixed in git. Unfortunately, I don't know that it is as\n> > > > trivial as just splitting the incoming stream; we would also have to\n> > > > make sure that there were no cross-pack deltas in the result.\n> > > \n> > > IMHO this is the wrong fix. The pack size limit was created to deal \n> > > with storage media with limited capacity. In this case, the repack \n> > > process should be told to limit its memory usage, and pack-index should \n> > > simply be taught to cope.\n> > \n> > Hmm, you're right. I was thinking it helped to deal with memory\n> > addressing issues for 32-bit systems, but I guess\n> > core.packedGitWindowSize should be handling that. IOW, the 10G packfile\n> > should work just fine for normal access.\n> > \n> > However, the OP did say he got an \"out of memory\" error during the\n> > clone. So maybe there is a problem to be fixed in index-pack there.\n> \n> Was the OOM on the remote side (pack-objects) or on the local side \n> (index-pack) ?\n> \n> Nicolas\n> \nTo be exact I did the clone locally on the same machine and so the clone itself \nworked\nbut I got the OOM during the first fetch. I \"fixed\" this by setting \ntransfer.unpacklimit=100000\nwhich caused only loose objects to be transfered. \nSo in this case I think the OOM was on the remote side. But there is another OOM \nif I try to repack locally.\nIt seems to me that neither pack-objects nor index-pack respekt \npack.packsizelimit and always\ntry to pack all objects to be transferred resp. all local loose objects in one \npack.\nI could live wth the transfer.unpacklimit=100000 but the local OOM stops me from \nusing the cloned repo.\n---\nThomas\n"},{"id":"191191","messageId":"CACsJy8BhSn+PB5tXME-w_cq4DVd2BULNRNLV-vk1_6yWKy+fNg@mail.gmail.com","threadId":"30405","inReplyTo":"loom.20120509T113505-740@post.gmane.org","subject":"Re: Large repo and pack.packsizelimit","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-05-09T10:50:55Z","receivedAt":"2012-05-09T10:50:55Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 9, 2012 at 4:36 PM, Thomas wrote:\n> To be exact I did the clone locally on the same machine and so the clone itself\n> worked\n> but I got the OOM during the first fetch. I \"fixed\" this by setting\n> transfer.unpacklimit=100000\n> which caused only loose objects to be transfered.\n> So in this case I think the OOM was on the remote side. But there is another OOM\n> if I try to repack locally.\n> It seems to me that neither pack-objects nor index-pack respekt\n> pack.packsizelimit and always\n> try to pack all objects to be transferred resp. all local loose objects in one\n> pack.\n> I could live wth the transfer.unpacklimit=100000 but the local OOM stops me from\n> using the cloned repo.\n\nI have some patches to make index-pack work better with large blobs\nbut they're not ready yet. I think pack-objects works fine with large\nblobs as long as they are all in packs. Are there any loose objects on\nthe source repo?\n\nIt's strange that you chose \"256mb\" as the upper limit for small\nobjects in your first mail. Do you have a lot of >=10mb files? By\ndefault, files smaller than 512mb will be put in memory for delta. A\nlot of big (but smaller than 512mb) files can quickly consume all\nmemory. If it's the case, maybe you can lower core.bigFileThreshold\n\nAlso maybe try remove the 1.2GB file from the source repo and see if\nit works better. That could give us some hints where the problem is.\n-- \nDuy\n"},{"id":"191192","messageId":"loom.20120509T131228-943@post.gmane.org","threadId":"30405","inReplyTo":"CACsJy8BhSn+PB5tXME-w_cq4DVd2BULNRNLV-vk1_6yWKy+fNg@mail.gmail.com","subject":"Re: Large repo and pack.packsizelimit","fromName":"Thomas","fromEmail":"th.acker66@arcor.de","sentAt":"2012-05-09T11:46:13Z","receivedAt":"2012-05-09T11:46:13Z","isPatch":false,"sender":{"key":"th.acker@arcor.de","avatar":"https://avatars.githubusercontent.com/u/1358536?v=4"},"body":"Nguyen Thai Ngoc Duy gmail.com> writes:\n\n> \n> On Wed, May 9, 2012 at 4:36 PM, Thomas arcor.de> wrote:\n> > To be exact I did the clone locally on the same machine and so the clone \nitself\n> > worked\n> > but I got the OOM during the first fetch. I \"fixed\" this by setting\n> > transfer.unpacklimit=100000\n> > which caused only loose objects to be transfered.\n> > So in this case I think the OOM was on the remote side. But there is another \nOOM\n> > if I try to repack locally.\n> > It seems to me that neither pack-objects nor index-pack respekt\n> > pack.packsizelimit and always\n> > try to pack all objects to be transferred resp. all local loose objects in \none\n> > pack.\n> > I could live wth the transfer.unpacklimit=100000 but the local OOM stops me \nfrom\n> > using the cloned repo.\n> \n> I have some patches to make index-pack work better with large blobs\n> but they're not ready yet. I think pack-objects works fine with large\n> blobs as long as they are all in packs. Are there any loose objects on\n> the source repo?\n> \n> It's strange that you chose \"256mb\" as the upper limit for small\n> objects in your first mail. Do you have a lot of >=10mb files? By\n> default, files smaller than 512mb will be put in memory for delta. A\n> lot of big (but smaller than 512mb) files can quickly consume all\n> memory. If it's the case, maybe you can lower core.bigFileThreshold\n> \n> Also maybe try remove the 1.2GB file from the source repo and see if\n> it works better. That could give us some hints where the problem is.\n\nI am using core.bigFileThreshold=256MB already; so the large file/s should not \nbe the problem (most of the files in the repo are \"standard\" source code files;\nI tried even smaller numbers for bigFileThreshold and packsizelimit but with no \nsuccess). \nAs long as I worked with the original repo which was updated regularily all \nworked well as soon as pack.packsizelimit was set to 1024MB (even with the 1.2GB \nfile). Repack seems not to increase a pack further as soon as packsizelimit is \nexceeded (so my packs are all slightly larger than 1024MB) BUT it also seems to \ntry to put everything in one pack regardless of packsizelimit in the following \ncases:\n(1) all objects to be transferred to another repo \n(2) all loose objects when starting a local repack\nCase (1) can be fixed by transfer.unpacklimit but there is no fix for (2).\n---\nThomas\n"},{"id":"191204","messageId":"7vlil1tg7w.fsf@alter.siamese.dyndns.org","threadId":"30405","inReplyTo":"loom.20120509T131228-943@post.gmane.org","subject":"Re: Large repo and pack.packsizelimit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-09T17:30:11Z","receivedAt":"2012-05-09T17:30:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas writes:\n\n> (1) all objects to be transferred to another repo \n> (2) all loose objects when starting a local repack\n> Case (1) can be fixed by transfer.unpacklimit but there is no fix for (2).\n\nTechnically (1) is putting everything in a single pack to transfer, and it\nis only the receiving end that does the chopping.\n\nFor (2), you could do something like\n\n\tkeep=$( git rev-list --objects $some_rev |\n\t git pack-objects --delta-base-offset pack ) &&\n\tmv pack-$keep.pack pack-$keep.idx .git/objects/pack/ &&\n\techo \"keep $some_rev\" >.git/objects/pack/pack-$keep.keep\n\nafter finding a suitable $some_rev that is old enough so that it will be\nan ancestor of anything that matters in the future and gives small enough\npackfiles. You may want to try doing the above multiple times, by picking\na few strategic ranges, e.g.\n\n\tfor some_rev in v1.0 v1.0..v2.0 v2.0..v2.4 v2.4..v3.0\n\tdo\n\t\t... the above four lines come here ...\n\tdone\n\nThe objects stored in .keep packs won't participate in future repack so\nyour \"git repack -a -d\" after that will put everything that is needed only\nfor versions newer than v3.0 in a single new pack.\n"},{"id":"191279","messageId":"loom.20120510T133639-197@post.gmane.org","threadId":"30405","inReplyTo":"7vlil1tg7w.fsf@alter.siamese.dyndns.org","subject":"Re: Large repo and pack.packsizelimit","fromName":"Thomas","fromEmail":"th.acker66@arcor.de","sentAt":"2012-05-10T11:42:09Z","receivedAt":"2012-05-10T11:42:09Z","isPatch":false,"sender":{"key":"th.acker@arcor.de","avatar":"https://avatars.githubusercontent.com/u/1358536?v=4"},"body":"Junio C Hamano pobox.com> writes:\n\n> \n> Thomas arcor.de> writes:\n> \n> > (1) all objects to be transferred to another repo \n> > (2) all loose objects when starting a local repack\n> > Case (1) can be fixed by transfer.unpacklimit but there is no fix for (2).\n> \n> Technically (1) is putting everything in a single pack to transfer, and it\n> is only the receiving end that does the chopping.\n> \n> For (2), you could do something like\n> \n> \tkeep=$( git rev-list --objects $some_rev |\n> \t git pack-objects --delta-base-offset pack ) &&\n> \tmv pack-$keep.pack pack-$keep.idx .git/objects/pack/ &&\n> \techo \"keep $some_rev\" >.git/objects/pack/pack-$keep.keep\n> \n> after finding a suitable $some_rev that is old enough so that it will be\n> an ancestor of anything that matters in the future and gives small enough\n> packfiles. You may want to try doing the above multiple times, by picking\n> a few strategic ranges, e.g.\n> \n> \tfor some_rev in v1.0 v1.0..v2.0 v2.0..v2.4 v2.4..v3.0\n> \tdo\n> \t\t... the above four lines come here ...\n> \tdone\n> \n\nNot really a porcelain-level solution ;-) but I will try it.\nThanks!\nIs there any chance that (1) and (2) will be solved by using packsizelimit\nin the future?\nWill there be any porcelain/plumbing commands for creating/deleting\n.keep files for packs?\n---\nThomas\n"}]}