{"thread":{"id":"12004","subject":"[RFC] repack vs re-clone","startedAt":"2008-02-10T08:25:49Z","lastAt":"2008-02-11T20:44:48Z","messageCount":9,"participants":["Marco Costalba","Nicolas Pitre","Jakub Narebski","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"68182","messageId":"e5bfff550802100025k616ccff5ib2917d283eeb0ff0@mail.gmail.com","threadId":"12004","inReplyTo":null,"subject":"[RFC] repack vs re-clone","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2008-02-10T08:25:49Z","receivedAt":"2008-02-10T08:25:49Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"Sometime I found myself re-cloning entirely a repository, as example\nthe Linux tree, instead of repackaging my local copy.\n\nThe reason is that the published Linux repository is super compressed\nand to reach the same level of compression on my local copy I would\nneed to give my laptop a long night running.\n\nSo it happens to be just faster to re-clone the whole thing by upstream.\n\nAlso repackaging a big repo in the optimal way is not so trivial, you\nneed to understand quite advanced stuff like window depth and so on\nand probably the pack parameters used upstream are easily better then\nwhat you could 'guess' trying yourself. Or simply you don't have\nenough RAM as would be needed.\n\nOn the other end it would be interesting to know, before to start the\nnew clone, what is the real advantage of this, i.e. what is the\nrepository size upstream.\n\nSo I would like to ask if anyone would consider useful:\n\n- A command like 'git info' or something like that that prints size of\nlocal and upstream repository (among possibly other things)\n\n- An option like 'git repack --clone' to instruct git to download and\nuse current upstream packs instead of trying to recreate new ones.\n\n\nMarco\n"},{"id":"68269","messageId":"alpine.LFD.1.00.0802101452550.2732@xanadu.home","threadId":"12004","inReplyTo":"e5bfff550802100025k616ccff5ib2917d283eeb0ff0@mail.gmail.com","subject":"Re: [RFC] repack vs re-clone","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-02-10T20:50:46Z","receivedAt":"2008-02-10T20:50:46Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 10 Feb 2008, Marco Costalba wrote:\n\n> Sometime I found myself re-cloning entirely a repository, as example\n> the Linux tree, instead of repackaging my local copy.\n> \n> The reason is that the published Linux repository is super compressed\n> and to reach the same level of compression on my local copy I would\n> need to give my laptop a long night running.\n\nNo. I really doubt the public Linux repository is compressed with \nanything but the default repack settings.\n\nAnd on my average PC by today's standards (P4 @ 3GHz with 1GB memory), \nrepacking the Linux repo takes less than 6.5 minutes, and peak RSS is \naround 450MB.\n\n> So it happens to be just faster to re-clone the whole thing by upstream.\n\nOnly if you're lucky to have a fast connection to the net with a high \ntransfer quota.\n\n> Also repackaging a big repo in the optimal way is not so trivial, you\n> need to understand quite advanced stuff like window depth and so on\n> and probably the pack parameters used upstream are easily better then\n> what you could 'guess' trying yourself. Or simply you don't have\n> enough RAM as would be needed.\n\nIf such is your case, why would you fully repack your repo in the first \nplace? Simply running 'git gc' should be quite good enough for people \nuninterested in the \"advanced\" stuff. The repack that 'git gc' uses \nwill happily reuse existing packed data from upstream.\n\n\n\n> On the other end it would be interesting to know, before to start the\n> new clone, what is the real advantage of this, i.e. what is the\n> repository size upstream.\n\nYou can already query the remote repository directory listing and figure \nit out. For example:\n\nlftp -c 'open http://kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/pack && ls'\n\nAnd you'll note that even upstream isn't always fully packed in advance.\n\n> So I would like to ask if anyone would consider useful:\n> \n> - A command like 'git info' or something like that that prints size of\n> local and upstream repository (among possibly other things)\n> \n> - An option like 'git repack --clone' to instruct git to download and\n> use current upstream packs instead of trying to recreate new ones.\n\nI think that would be a very bad idea. Not only this is rather \nunnecessary (either you can aford to repack locally, or you live with \nthe upstream provided packing and repack incrementally which is pretty \ngood enough), but it is also really bad resource wise (that'll end up \nonly wasting net bandwitdh and CPU cycles on the server).\n\n\nNicolas\n"},{"id":"68419","messageId":"m3myq7e3ie.fsf@localhost.localdomain","threadId":"12004","inReplyTo":"e5bfff550802100025k616ccff5ib2917d283eeb0ff0@mail.gmail.com","subject":"Re: [RFC] repack vs re-clone","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-02-11T18:45:24Z","receivedAt":"2008-02-11T18:45:24Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Marco Costalba\" writes:\n\n> Sometime I found myself re-cloning entirely a repository, as example\n> the Linux tree, instead of repackaging my local copy.\n> \n> The reason is that the published Linux repository is super compressed\n> and to reach the same level of compression on my local copy I would\n> need to give my laptop a long night running.\n\nRepacking without '--force' (for gc) or '--no-reuse-delta' (low level)\nwould always reuse this tight packing. Only with '--force' you would\nwaste CPU trying to find better deltaification[*1*].\n \n> So it happens to be just faster to re-clone the whole thing by upstream.\n\nSo what you are doing is passing the work, unnecessary work I'd say,\nto some poor server. Not nice.\n\n\n[*1*] I hope that '--no-reuse-delta' means _try_ to find better delta,\nbut use current one as possible delta, not stupid forget about current\ndeltaification at all...\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"68424","messageId":"e5bfff550802111120x24338cd6n6eb1f6db55fe487f@mail.gmail.com","threadId":"12004","inReplyTo":"m3myq7e3ie.fsf@localhost.localdomain","subject":"Re: [RFC] repack vs re-clone","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2008-02-11T19:20:09Z","receivedAt":"2008-02-11T19:20:09Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Feb 11, 2008 7:45 PM, Jakub Narebski wrote:\n> \"Marco Costalba\" writes:\n>\n> > So it happens to be just faster to re-clone the whole thing by upstream.\n>\n> So what you are doing is passing the work, unnecessary work I'd say,\n> to some poor server. Not nice.\n>\n\nTo a poor net bandwidth I would say because cloning from zero just\ndownloads the packages.\n"},{"id":"68427","messageId":"alpine.LFD.1.00.0802111436100.2732@xanadu.home","threadId":"12004","inReplyTo":"m3myq7e3ie.fsf@localhost.localdomain","subject":"Re: [RFC] repack vs re-clone","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-02-11T19:40:00Z","receivedAt":"2008-02-11T19:40:00Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 11 Feb 2008, Jakub Narebski wrote:\n\n> [*1*] I hope that '--no-reuse-delta' means _try_ to find better delta,\n> but use current one as possible delta, not stupid forget about current\n> deltaification at all...\n\nIt is really \"forget about everything\". And by the time you look for \nthe best delta from scratch, remembering what was the best delta before \nwon't give you much performance gain, plus it has nasty issues like \nmaking sure no delta cycles are created if you reuse an old delta, etc.\n\n\nNicolas\n"},{"id":"68430","messageId":"alpine.LSU.1.00.0802111950000.3870@racer.site","threadId":"12004","inReplyTo":"e5bfff550802111120x24338cd6n6eb1f6db55fe487f@mail.gmail.com","subject":"Re: [RFC] repack vs re-clone","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-11T19:50:47Z","receivedAt":"2008-02-11T19:50:47Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 11 Feb 2008, Marco Costalba wrote:\n\n> On Feb 11, 2008 7:45 PM, Jakub Narebski wrote:\n> > \"Marco Costalba\" writes:\n> >\n> > > So it happens to be just faster to re-clone the whole thing by \n> > > upstream.\n> >\n> > So what you are doing is passing the work, unnecessary work I'd say, \n> > to some poor server. Not nice.\n> \n> To a poor net bandwidth I would say because cloning from zero just \n> downloads the packages.\n\nNot exactly. Remember, if you did not \"repack\" via clone, the server \nwould spend less cycles _and_ bandwidth on you. Namely zero.\n\nCiao,\nDscho\n"},{"id":"68431","messageId":"200802112051.27271.jnareb@gmail.com","threadId":"12004","inReplyTo":"e5bfff550802111120x24338cd6n6eb1f6db55fe487f@mail.gmail.com","subject":"Re: [RFC] repack vs re-clone","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-02-11T19:51:26Z","receivedAt":"2008-02-11T19:51:26Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Marco Costalba wrote:\n> On Feb 11, 2008 7:45 PM, Jakub Narebski wrote:\n> > \"Marco Costalba\" writes:\n> >\n> > > So it happens to be just faster to re-clone the whole thing by upstream.\n> >\n> > So what you are doing is passing the work, unnecessary work I'd say,\n> > to some poor server. Not nice.\n> \n> To a poor net bandwidth I would say because cloning from zero just\n> downloads the packages.\n\nCloning from zero over http, https and rsync (and ftp) just downloads\nthe packfiles. Cloning over git or ssh if I understand correctly[*1*]\ngenerates single pack for transfer. And that generates load for server.\n\n[*1*] If I undersnad correctly from discussions here on git mailing\n list, the pack transfer protocol currently can transfer only\n _single_ pack; proposed multi-pack extension didn't get\n implemented.\n-- \nJakub Narebski\nPoland\n"},{"id":"68432","messageId":"200802112053.35141.jnareb@gmail.com","threadId":"12004","inReplyTo":"alpine.LFD.1.00.0802111436100.2732@xanadu.home","subject":"Re: [RFC] repack vs re-clone","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-02-11T19:53:34Z","receivedAt":"2008-02-11T19:53:34Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nicolas Pitre wrote:\n> On Mon, 11 Feb 2008, Jakub Narebski wrote:\n> \n> > [*1*] I hope that '--no-reuse-delta' means _try_ to find better delta,\n> > but use current one as possible delta, not stupid forget about current\n> > deltaification at all...\n> \n> It is really \"forget about everything\". And by the time you look for \n> the best delta from scratch, remembering what was the best delta before \n> won't give you much performance gain, plus it has nasty issues like \n> making sure no delta cycles are created if you reuse an old delta, etc.\n\nSo we have either: don't try to find better delta (default), or forget\ntotally about old delta (--no-reuse-delta), no middle ground? That's\nbad...\n\nOr the default is try to find better delta, but reuse if better?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"68438","messageId":"alpine.LFD.1.00.0802111542000.2732@xanadu.home","threadId":"12004","inReplyTo":"200802112051.27271.jnareb@gmail.com","subject":"Re: [RFC] repack vs re-clone","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-02-11T20:44:48Z","receivedAt":"2008-02-11T20:44:48Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 11 Feb 2008, Jakub Narebski wrote:\n\n> Marco Costalba wrote:\n> > On Feb 11, 2008 7:45 PM, Jakub Narebski wrote:\n> > > \"Marco Costalba\" writes:\n> > >\n> > > > So it happens to be just faster to re-clone the whole thing by upstream.\n> > >\n> > > So what you are doing is passing the work, unnecessary work I'd say,\n> > > to some poor server. Not nice.\n> > \n> > To a poor net bandwidth I would say because cloning from zero just\n> > downloads the packages.\n> \n> Cloning from zero over http, https and rsync (and ftp) just downloads\n> the packfiles. Cloning over git or ssh if I understand correctly[*1*]\n> generates single pack for transfer. And that generates load for server.\n\nThe created pack will always reuse existing deltas, so the load is more \nabout making sure the sent pack contains only needed objects for the \nrequired branch -- something that dumb protocols cannot do.\n\n\nNicolas\n"}]}