{"thread":{"id":"26202","subject":"git fails on large repo clone on intermittent, or intermittently-high-latency, connections","startedAt":"2011-01-05T14:28:40Z","lastAt":"2011-01-09T20:04:55Z","messageCount":5,"participants":["Zenaan Harkness","Jakub Narebski","Jonathan Nieder","James Cloos"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"158960","messageId":"AANLkTinkhmHpAQjraviKjqFAczThR5GtT_qdyf6Sb3nm@mail.gmail.com","threadId":"26202","inReplyTo":null,"subject":"git fails on large repo clone on intermittent, or intermittently-high-latency, connections","fromName":"Zenaan Harkness","fromEmail":"zen@freedbms.net","sentAt":"2011-01-05T14:28:40Z","receivedAt":"2011-01-05T14:28:40Z","isPatch":false,"sender":{"key":"zen@freedbms.net","avatar":null},"body":"Hi, I am trying to clone opentaps.git. The following is my third try,\nand I am giving up now. As you can see I pressed <Return> every now\nand then, and for the last long while, absolutely nothing downloading\n- the connection has clearly died once again. My last run was\nyesterday and I left it run overnight.\n\nI cannot download (with a git clone/ initial repo), more than roughly\n100MiB. I am on a satellite connection. I have also experienced this\nwith wireless connection about 18 months ago.\n\nI have in the middle there (a few months ago) spent a couple months at\na friend's place, and never had the same problem - nice ADSL2+\nconnection, ~1.5MiB/s connections. As you can see with my satellite\n(rural) and also with my older wireless (also rural) connections, I do\nnot get more than about 64KiB/s, but it's usually slower, and I've\nalways had satellite latency issues in the order of 450ms, and\nsometimes the odd dropout.\n\nAs opposed to git, wget on the other hand not only retries and\ncontinues from where it left off when it retries (HTTP protocol)\n(default retry 10 times I think), but I can completely INTerrupt wget,\nand start it again from an entirely different computer if I want\n(using wget's --continue option), and it will (with HTTP) happily\ncontinue right where it left off, and eventually my download\ncompletes.\n\nNOT so with git! :\n\n$ git clone git://gitorious.org/opentaps/opentaps.git opentaps.git\nCloning into opentaps.git...\nremote: Counting objects: 105724, done.\nremote: Compressing objects: 100% (30417/30417), done.\nReceiving objects:   5% (5888/105724), 10.44 MiB | 21 KiB/s\nReceiving objects:   5% (5898/105724), 12.18 MiB | 51 KiB/s\nReceiving objects:   5% (5920/105724), 17.47 MiB | 38 KiB/s\nReceiving objects:   5% (5923/105724), 19.64 MiB | 23 KiB/s\nReceiving objects:   5% (5939/105724), 30.01 MiB | 27 KiB/s\nReceiving objects:   5% (6184/105724), 41.00 MiB | 47 KiB/s\nReceiving objects:   7% (7818/105724), 52.77 MiB | 58 KiB/s\nReceiving objects:   8% (9170/105724), 67.66 MiB | 56 KiB/s\nReceiving objects:  10% (11309/105724), 70.57 MiB | 24 KiB/s\nReceiving objects:  12% (13413/105724), 82.43 MiB | 29 KiB/s\nReceiving objects:  12% (13495/105724), 96.81 MiB | 39 KiB/s\nReceiving objects:  12% (13495/105724), 101.57 MiB | 47 KiB/s\nReceiving objects:  12% (13523/105724), 142.64 MiB | 27 KiB/s\n<here it died, after over an hour dead, I killed it completely>\n\nGit cannot operate robustly with larger repos, it appears to me, on\ninternet connections with even slightly flaky links.\n\nI've googled for a tar-ball of the git repo for opentaps, but found nothing.\n\nWhat can I do to work around my flaky link?\n\nHow hard would it be to add a wget-like mode to git, for the initial\nrepo download?\n\nTIA\nZen\n"},{"id":"158963","messageId":"m3tyhnbcf7.fsf@localhost.localdomain","threadId":"26202","inReplyTo":"AANLkTinkhmHpAQjraviKjqFAczThR5GtT_qdyf6Sb3nm@mail.gmail.com","subject":"Re: git fails on large repo clone on intermittent, or intermittently-high-latency, connections","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-01-05T15:26:31Z","receivedAt":"2011-01-05T15:26:31Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Zenaan Harkness <zen@freedbms.net> writes:\n\n> Hi, I am trying to clone opentaps.git. The following is my third try,\n> and I am giving up now. As you can see I pressed <Return> every now\n> and then, and for the last long while, absolutely nothing downloading\n> - the connection has clearly died once again. My last run was\n> yesterday and I left it run overnight.\n> \n> I cannot download (with a git clone/ initial repo), more than roughly\n> 100MiB. I am on a satellite connection. I have also experienced this\n> with wireless connection about 18 months ago.\n> \n> I have in the middle there (a few months ago) spent a couple months at\n> a friend's place, and never had the same problem - nice ADSL2+\n> connection, ~1.5MiB/s connections. As you can see with my satellite\n> (rural) and also with my older wireless (also rural) connections, I do\n> not get more than about 64KiB/s, but it's usually slower, and I've\n> always had satellite latency issues in the order of 450ms, and\n> sometimes the odd dropout.\n> \n> As opposed to git, wget on the other hand not only retries and\n> continues from where it left off when it retries (HTTP protocol)\n> (default retry 10 times I think), but I can completely INTerrupt wget,\n> and start it again from an entirely different computer if I want\n> (using wget's --continue option), and it will (with HTTP) happily\n> continue right where it left off, and eventually my download\n> completes.\n\n[...]\n> What can I do to work around my flaky link?\n\nAsk project in question to provide bundle of repository for seeding\ninitial clone (see git-bundle manpage); this is an ordinary file, and\ncan be downloaded via HTTP or even P2P.\n\n> \n> How hard would it be to add a wget-like mode to git, for the initial\n> repo download?\n\nVery hard; tthough \"resumable clone\" was often requested (25%\nresponders in \"Git User's Survey 2010\", see [1]), and there was even\nsome discussion about possible implementation, it was not implemented\nyet, even as proof of concept.\n\nThe trouble is that packfile is *generated for a client*, and\nbit-for-bit representation of said pack can vary (e.g. if\nmultithreaded packing is enabled; usually a good idea).\n\n[1]: https://git.wiki.kernel.org/index.php/GitSurvey2010#17._Which_of_the_following_features_would_you_like_to_see_implemented_in_git.3F\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"158973","messageId":"20110105175412.GA21863@burratino","threadId":"26202","inReplyTo":"m3tyhnbcf7.fsf@localhost.localdomain","subject":"Re: git fails on large repo clone on intermittent, or intermittently-high-latency, connections","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-05T17:54:12Z","receivedAt":"2011-01-05T17:54:12Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jakub Narebski wrote:\n> Zenaan Harkness <zen@freedbms.net> writes:\n\n>> How hard would it be to add a wget-like mode to git, for the initial\n>> repo download?\n>\n> Very hard; tthough \"resumable clone\" was often requested (25%\n> responders in \"Git User's Survey 2010\", see [1]), and there was even\n> some discussion about possible implementation, it was not implemented\n> yet, even as proof of concept.\n>\n> The trouble is that packfile is *generated for a client*, and\n> bit-for-bit representation of said pack can vary (e.g. if\n> multithreaded packing is enabled; usually a good idea).\n\nThat said, one possible partial solution would be to automate\ngeneration of a seed bundle for huge repositories (with a script or\na special parameter to \"git gc\", maybe) and to document serving such a\nseed bundle over HTTP as part of the standard setup.  If this could be\nmade simple enough that e.g. all large repos on repo.or.cz had such a\nseed bundle then I would call it a success. :)\n"},{"id":"158984","messageId":"201101052100.49506.jnareb@gmail.com","threadId":"26202","inReplyTo":"20110105175412.GA21863@burratino","subject":"Re: git fails on large repo clone on intermittent, or intermittently-high-latency, connections","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-01-05T20:00:48Z","receivedAt":"2011-01-05T20:00:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 5 Jan 2011, Jonathan Nieder wrote:\n> Jakub Narebski wrote:\n> > Zenaan Harkness <zen@freedbms.net> writes:\n> \n> > > How hard would it be to add a wget-like mode to git, for the initial\n> > > repo download?\n> >\n> > Very hard; tthough \"resumable clone\" was often requested (25%\n> > responders in \"Git User's Survey 2010\", see [1]), and there was even\n> > some discussion about possible implementation, it was not implemented\n> > yet, even as proof of concept.\n> >\n> > The trouble is that packfile is *generated for a client*, and\n> > bit-for-bit representation of said pack can vary (e.g. if\n> > multithreaded packing is enabled; usually a good idea).\n> \n> That said, one possible partial solution would be to automate\n> generation of a seed bundle for huge repositories (with a script or\n> a special parameter to \"git gc\", maybe) and to document serving such a\n> seed bundle over HTTP as part of the standard setup.  If this could be\n> made simple enough that e.g. all large repos on repo.or.cz had such a\n> seed bundle then I would call it a success. :)\n\nI wonder if adding support for per-project _bundle_ link and 'bundle'\naction support to gitweb (perhaps only if caching is turned on) would\nhelp there... though I am not sure if doenloading fron gitweb is\nresumable.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"159260","messageId":"m38vyt7shs.fsf@jhcloos.com","threadId":"26202","inReplyTo":"AANLkTinkhmHpAQjraviKjqFAczThR5GtT_qdyf6Sb3nm@mail.gmail.com","subject":"Re: git fails on large repo clone on intermittent, or intermittently-high-latency, connections","fromName":"James Cloos","fromEmail":"cloos@jhcloos.com","sentAt":"2011-01-09T20:04:55Z","receivedAt":"2011-01-09T20:04:55Z","isPatch":false,"sender":{"key":"cloos@jhcloos.com","avatar":"https://gravatar.com/avatar/ec9a05787d29afe41e243e4b60bd0e2f69d757688e8f0bfe5e78bc185a3e317f?d=mp&s=160"},"body":"In addition to the other replies, if you have a shell login elswhere you\ncan clone there, bundle the file, and use rsync, http, ftp or the like\nto copy it down.\n\nIf the remote site's git is too old to have git bundle, use a bare clone\nand tar it.  You will not need to compress the tar.\n\nYou can also use split(1) to break up the bundle or tar into smaller\nchunks if that helps.  cat(1) will happily recombine those chunks.\n\nIf git bundle was not available, you can use the copied bare repo as\na --reference for a new clone, then copy the bare's pack file into\nthat new clone and remove the new clone's objects/info/alternates file.\n\nI've had to use that method to get a clean clone across a small straw\n(dialup or wireless) for several large repositories over the years.\n\n-JimC\n-- \nJames Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6\n"}]}