{"thread":{"id":"24919","subject":"Large pack causes git clone failures ... what to do?","startedAt":"2010-08-31T07:16:58Z","lastAt":"2010-09-06T00:34:55Z","messageCount":7,"participants":["Geoff Russell","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"149397","messageId":"AANLkTi=1iLx=-9gxkGzuhrbpA005VPSp0itkAkOG4D4z@mail.gmail.com","threadId":"24919","inReplyTo":null,"subject":"Large pack causes git clone failures ... what to do?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2010-08-31T07:16:58Z","receivedAt":"2010-08-31T07:16:58Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Hi,\n\nI did a \"git gc\" on a repository and ended up with a 4GB pack ... now I\ncan't clone the repository and get the following:\n\n\nremote: fatal: Out of memory? mmap failed: Cannot allocate memory\nremote: aborting due to possible repository corruption on the remote side.\nfatal: early EOF\nerror: git upload-pack: git-pack-objects died with error.\nfatal: git upload-pack: aborting due to possible repository corruption\non the remote side.\nfatal: index-pack failed\n\nHow do I deal with this? I'm running git version 1.6.2.3\n\nI've looked at \"git repack --max-pack-size\", but which that created new packs it\ndidn't delete the old monster. If I run gc, how do I tell it about the\nmax-pack size? It doesn't\nseem to support this argument.\n\nCheers,\nGeoff\n\n--\n6 Fifth Ave,\nSt Morris, S.A. 5068\nAustralia\nPh: 041 8805 184 / 08 8332 5069\nhttp://perfidy.com.au\n"},{"id":"149464","messageId":"20100831180247.GF32601@spearce.org","threadId":"24919","inReplyTo":"AANLkTi=1iLx=-9gxkGzuhrbpA005VPSp0itkAkOG4D4z@mail.gmail.com","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-08-31T18:02:47Z","receivedAt":"2010-08-31T18:02:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Geoff Russell wrote:\n> I did a \"git gc\" on a repository and ended up with a 4GB pack ... now I\n> can't clone the repository and get the following:\n> \n> remote: fatal: Out of memory? mmap failed: Cannot allocate memory\n> remote: aborting due to possible repository corruption on the remote side.\n> fatal: early EOF\n> error: git upload-pack: git-pack-objects died with error.\n> fatal: git upload-pack: aborting due to possible repository corruption\n> on the remote side.\n> fatal: index-pack failed\n> \n> How do I deal with this? I'm running git version 1.6.2.3\n\nAre you on a 32 bit Linux system? Or 64 bit? Git should be auto\nselecting a unit that would allow it to mmap slices of that 4GB pack.\n\n> I've looked at \"git repack --max-pack-size\", but which that\n> created new packs it didn't delete the old monster.\n\nYou really needed to run:\n\n git repack --max-pack-size=.. -a -d\n\nThe -d flag tells it to remove the old packs once the new packs\nare ready, and the -a flag tells it to reconsider every object\nin the repository, rather than just those that are loose.\n\nBut if you can't clone it, you probably can't repack it. Clone works\nby creating a pack file on the server, just like repack does.\nExcept it sends the pack out to the network stream instead of to\nlocal disk.\n\n-- \nShawn.\n"},{"id":"149502","messageId":"AANLkTi=O5SbLRttzR0YwrHVEMz5gxtdTo9Z5C6V1yE1e@mail.gmail.com","threadId":"24919","inReplyTo":"20100831180247.GF32601@spearce.org","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2010-08-31T22:03:14Z","receivedAt":"2010-08-31T22:03:14Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Thanks Shawn,\n\nOn Wed, Sep 1, 2010 at 3:32 AM, Shawn O. Pearce wrote:\n> Geoff Russell wrote:\n>> I did a \"git gc\" on a repository and ended up with a 4GB pack ... now I\n>> can't clone the repository and get the following:\n>> ...\n>\n> Are you on a 32 bit Linux system?  Or 64 bit?  Git should be auto\n> selecting a unit that would allow it to mmap slices of that 4GB pack.\n\n32bit\n\n>\n>> I've looked at \"git repack --max-pack-size\", but which that\n>> created new packs it didn't delete the old monster.\n>\n> You really needed to run:\n>\n>  git repack --max-pack-size=.. -a -d\n>\n> The -d flag tells it to remove the old packs once the new packs\n> are ready, and the -a flag tells it to reconsider every object\n> in the repository, rather than just those that are loose.\n\nOk, will try.\n\n>\n> But if you can't clone it, you probably can't repack it.  Clone works\n\nThe cloning fails at different points in the process and the server is normally\nunder some load, so perhaps load is a factor.\n\n> by creating a pack file on the server, just like repack does.\n> Except it sends the pack out to the network stream instead of to\n> local disk.\n\nDoes clone from a client take note of the pack.packSizeLimit if I set it\non the server? Or does it use the client value?\n\nCheers and many thanks, annoying problems like this always happen at really\ninconvenient times :)\n\nGeoff.\n\n>\n> --\n> Shawn.\n>\n"},{"id":"149515","messageId":"AANLkTik2Ms6qjQAaa_H8GGwmTG1fsk+8GR2y7ZmLviY-@mail.gmail.com","threadId":"24919","inReplyTo":"AANLkTi=O5SbLRttzR0YwrHVEMz5gxtdTo9Z5C6V1yE1e@mail.gmail.com","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2010-09-01T01:53:23Z","receivedAt":"2010-09-01T01:53:23Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"On Wed, Sep 1, 2010 at 7:33 AM, Geoff Russell\n wrote:\n> Thanks Shawn,\n>\n>...\n>> You really needed to run:\n>>\n>>  git repack --max-pack-size=.. -a -d\n>>\n>> The -d flag tells it to remove the old packs once the new packs\n>> are ready, and the -a flag tells it to reconsider every object\n>> in the repository, rather than just those that are loose.\n>\n> Ok, will try.\n\nThe repack failed with a \"fatal: Out of memory, malloc failed\", perhaps I\njust need to try a machine with more memory!\n\nI'm still interested in whether clone from a client take note of the\npack.packSizeLimit if I set it\non the server? Or does it use the client value?\n\nCheers,\nGeoff\n"},{"id":"149521","messageId":"AANLkTikvkuCjm2+EmGLkWkucZrqxMH8vL9TBymfRk=0e@mail.gmail.com","threadId":"24919","inReplyTo":"AANLkTik2Ms6qjQAaa_H8GGwmTG1fsk+8GR2y7ZmLviY-@mail.gmail.com","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2010-09-01T03:02:58Z","receivedAt":"2010-09-01T03:02:58Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"On Wed, Sep 1, 2010 at 11:23 AM, Geoff Russell\n wrote:\n> On Wed, Sep 1, 2010 at 7:33 AM, Geoff Russell\n> wrote:\n>> Thanks Shawn,\n>>\n>>...\n>>> You really needed to run:\n>>>\n>>>  git repack --max-pack-size=.. -a -d\n>>>\n>>> The -d flag tells it to remove the old packs once the new packs\n>>> are ready, and the -a flag tells it to reconsider every object\n>>> in the repository, rather than just those that are loose.\n>>\n>> Ok, will try.\n>\n> The repack failed with a \"fatal: Out of memory, malloc failed\", perhaps I\n> just need to try a machine with more memory!\n\nOk, I rsynced the directory to a machine with 12Gb of memory and ran the\nrepack (git version 1.7.2.2) the repack worked (and quickly) but left\na \"bad\" sha1 file\nbehind:\n\n$ git repack --max-pack-size=100M -a -d\nCounting objects: 517563, done.\nDelta compression using up to 8 threads.\nCompressing objects: 100% (154217/154217), done.\nWriting objects: 100% (517563/517563), done.\nTotal 517563 (delta 353081), reused 465715 (delta 335261)\nRemoving duplicate objects: 100% (256/256), done.\n\n$ git fsck\nbad sha1 file: ./objects/5b/.fd25f132c21493b661978fc9362f673ea6e58b.cwxzjT\ndangling commit c7a4ecaa1732869f9bfa21d948cb8714fd303713\n\nI removed the bad file on the presumption that it was a working file\nand reran the fsck and all looked okay.\n\nCheers,\nGeoff.\n"},{"id":"149560","messageId":"20100901143819.GI32601@spearce.org","threadId":"24919","inReplyTo":"AANLkTik2Ms6qjQAaa_H8GGwmTG1fsk+8GR2y7ZmLviY-@mail.gmail.com","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-09-01T14:38:19Z","receivedAt":"2010-09-01T14:38:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Geoff Russell wrote:\n> \n> I'm still interested in whether clone from a client take note of the\n> pack.packSizeLimit if I set it\n> on the server? Or does it use the client value?\n\nNeither.\n\nA clone doesn't split its pack. It stores the entire project\nas a single pack file. If your filesystem cannot do that, the\nclone fails.\n\n-- \nShawn.\n"},{"id":"150020","messageId":"AANLkTi=4vF5o+oU-s=j8AyLjZ-FL5i2CbVv-ZQmkiB+N@mail.gmail.com","threadId":"24919","inReplyTo":"20100901143819.GI32601@spearce.org","subject":"Re: Large pack causes git clone failures ... what to do?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2010-09-06T00:34:55Z","receivedAt":"2010-09-06T00:34:55Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"On Thu, Sep 2, 2010 at 12:08 AM, Shawn O. Pearce wrote:\n> Geoff Russell wrote:\n>>\n>> I'm still interested in whether clone from a client take note of the\n>> pack.packSizeLimit if I set it\n>> on the server? Or does it use the client value?\n>\n> Neither.\n>\n> A clone doesn't split its pack.  It stores the entire project\n> as a single pack file.  If your filesystem cannot do that, the\n> clone fails.\n\nI've moved the \"master\" repository to a faster machine with plenty of\nmemory and all the problems have gone away. I was making wrong\nguesses about the cause. A fresh clone gives a huge pack, but no problems\nand everything runs much better\n\nThanks for your help.\n\nGeoff.\n"}]}