{"thread":{"id":"2283","subject":"git push sends more objects than it needs to","startedAt":"2005-10-31T18:23:55Z","lastAt":"2005-10-31T19:50:48Z","messageCount":5,"participants":["Luck, Tony","Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10862","messageId":"20051031182355.GA7368@agluck-lia64.sc.intel.com","threadId":"2283","inReplyTo":null,"subject":"git push sends more objects than it needs to","fromName":"Luck, Tony","fromEmail":"tony.luck@intel.com","sentAt":"2005-10-31T18:23:55Z","receivedAt":"2005-10-31T18:23:55Z","isPatch":false,"sender":{"key":"tony.luck@intel.com","avatar":"https://avatars.githubusercontent.com/u/5446021?v=4"},"body":"I just pulled from Linus' Linux repository ... and my\nwrapper script alerted me to the fact that he had just\nre-packed.\n\nSo, I logged into kernel.org and linked the new packfiles\nacross to my repo there, and did a \"git prune-packed\" to\nclean away all the unneeded unpacked objects.\n\nBack on my local box I merged all the new stuff that\nhad been pulled into my \"release\" and \"test\" branches.\n(just a fast-forward for the release branch).\n\nNext I used \"scp\" to copy Linus' new pack files to my\nlocal objects/pack ... and ran git prune packed here.\n\nFinally, after making sure that release and test\nbranches still built with all the new stuff I did\na \"git push\" to update my kernel repo. I saw this:\nupdating 'refs/heads/test'\n from 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f\n to 2e488d3f8f299eb0ed826055bc30003e2c27d909\nupdating 'refs/heads/release'\n from 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404\n to 581c1b14394aee60aff46ea67d05483261ed6527\nPacking 10785 objects\nUnpacking 10785 objects\n 100% (10785/10785) done\nrefs/heads/test: 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f -> 2e488d3f8f299eb0ed826055bc30003e2c27d909\nrefs/heads/release: 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404 -> 581c1b14394aee60aff46ea67d05483261ed6527\n\nNow the \"unpack\" on kernel.org did the right thing and noticed\nthat over 9000 of the objects were already in the packfile. But\nI wonder if it couldn't have been smarter and not sent them?\n\nOr am I just subverting the whole paradigm by hand-copying\npackfiles around?\n\n-Tony\n"},{"id":"10864","messageId":"Pine.LNX.4.64.0510311035310.27915@g5.osdl.org","threadId":"2283","inReplyTo":"20051031182355.GA7368@agluck-lia64.sc.intel.com","subject":"Re: git push sends more objects than it needs to","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-31T18:44:02Z","receivedAt":"2005-10-31T18:44:02Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 31 Oct 2005, Luck, Tony wrote:\n> \n> Finally, after making sure that release and test\n> branches still built with all the new stuff I did\n> a \"git push\" to update my kernel repo. I saw this:\n...\n> Packing 10785 objects\n> Unpacking 10785 objects\n> \n> Now the \"unpack\" on kernel.org did the right thing and noticed\n> that over 9000 of the objects were already in the packfile. But\n> I wonder if it couldn't have been smarter and not sent them?\n\nIt should have been smarter, but I suspect you got caught by the fact that \nkernel.org by default has git-0.99.8f on it, which has the old \npre-multi_ack code to figure out what the common commit was.\n\nThe pack-file contains over ten times as many objects, so you definitely \ndidn't get all of them - but because there has been a lot of merges \nlately, and the common commit finder algorithm wasn't all that careful, \nyou ended up getting many more objects than you really needed.\n\nRemember: the pack generation is not \"exact\" - it will often generate a \nfew extra objects for any non-trivial case (for example, it fundamentally \nhappens if there has been reverts: it won't realize that you had the older \nversion of a file already). It just so happens that the old algorithm had \nsome cases where it would decide on totally the wrong common commit, and \nre-send a _lot: more objects than it needs.\n\nI'm not sure multi-ack fixes it entirely either, but I think it makes it a \nlot less likely (but even with multi-ack, the \"file revert\" case still \nhappens, so you should always expect that can get a _couple_ of \nunnecessary objects).\n\nNow, even the old stupid algorithm got the _easy_ cases obviously right, \nso people might have incorrectly gotten the idea that it was careful and \nexact, just because quite often it ends up being that in practice.\n\n\t\tLinus\n"},{"id":"10865","messageId":"7vbr15iksc.fsf@assigned-by-dhcp.cox.net","threadId":"2283","inReplyTo":"20051031182355.GA7368@agluck-lia64.sc.intel.com","subject":"Re: git push sends more objects than it needs to","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-31T18:49:39Z","receivedAt":"2005-10-31T18:49:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Luck, Tony\" writes:\n\n> Now the \"unpack\" on kernel.org did the right thing and noticed\n> that over 9000 of the objects were already in the packfile. But\n> I wonder if it couldn't have been smarter and not sent them?\n>\n> Or am I just subverting the whole paradigm by hand-copying\n> packfiles around?\n\nThe fix for this problem is in 0.99.9 (credits to Johannes).\nUnfortunately, 0.99.8f is still in /usr/bin/ on kernel.org.\n"},{"id":"10868","messageId":"Pine.LNX.4.64.0510311119170.27915@g5.osdl.org","threadId":"2283","inReplyTo":"Pine.LNX.4.64.0510311035310.27915@g5.osdl.org","subject":"Re: git push sends more objects than it needs to","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-31T19:36:09Z","receivedAt":"2005-10-31T19:36:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 31 Oct 2005, Linus Torvalds wrote:\n> > \n> > Now the \"unpack\" on kernel.org did the right thing and noticed\n> > that over 9000 of the objects were already in the packfile. But\n> > I wonder if it couldn't have been smarter and not sent them?\n> \n> It should have been smarter, but I suspect you got caught by the fact that \n> kernel.org by default has git-0.99.8f on it, which has the old \n> pre-multi_ack code to figure out what the common commit was.\n\nOh, actually, I take that back.\n\nThis is not the multi-ack code at all, I suspect.\n\nThe problem is totally different: you copied the new pack to your \nmaster.kernel.org repository, but you never updated any branches there.\n\nSo you had the objects, but git had no way of knowing. The pack generation \ndoesn't look at what _objects_ you have, it looks at what _refs_ you have. \nAnd your refs were all to the old state.\n\nSo git actually did everything right (you can never rely on objects: you \nmay have a partial object list due to some earlier incomplete pull/push). \n\nThis is actually easy enough to fix up in one of several ways:\n\n a) The \"don't do that then\" approach:\n\n Don't go behind git's back and add objects on your own, and expect git \n to realize what you did. ;^p\n\n b) The \"live with it\" approach:\n\n You copied the pack by hand, and that will keep git-unpack-objects \n from duplicating the objects, but you'll still waste time and network \n when trying pushing the objects (just once, though).\n\n Ie this is what happened this time: nothing really lost, and the end \n result is fine. Now you know why it happened, and you're fine.\n\n c) The \"I'm smarter than git\" approach:\n\n When you copy my objects, copy my reference to the top-of-tree too \n (and rename it). NOTE! Now you need to be really careful, and you need \n to make sure you copy _all_ the objects, because if you screw this up, \n your repo will be missing objects that you claim are there, and it \n will be all your fault.\n\n I really don't advocate this approach at all. It's certainly doable, \n but it's also the only approach where you can really screw up.\n\n d) Just let git do it for you.\n\n Copy the pack-files, or add my object directory as an \"alternates\" for \n your object directory, do the \"git prune-objects\", and then _locally_ \n on master.kernel.org just do something like\n\n\tgit fetch ..linus-directory.. master:linus\n\n which will still create the unnecessary pack-file and unpack it into \n nothingness (since you have the objects in the pack-file you copied by \n hand), but at least it won't eat any network bandwidth, and it will do \n the right thing if it turns out that I've pushed something after doing \n the pack-file, and fetch those individual objects in _addition_ to the \n pack-file you snarfed by hand. It will also obviously update a ref in\n your tree (the \"linus\" branch), so now when you send stuff later, it \n will know all about the objects you already have.\n\n e) Re-create the tree entirely\n\n Blow away your tree on master entirely, just re-create it locally with \n \"git clone -l -s\" from my tree (which will do all the \"alternates\" \n object files for you), and then populate the result with a simple push\n from your home tree.\n\n f) any number of variations on a theme. IOW, there are endless ways to do \n this. \n\nHmm?\n\n\t\tLinus\n"},{"id":"10871","messageId":"20051031195048.GA9231@agluck-lia64.sc.intel.com","threadId":"2283","inReplyTo":"Pine.LNX.4.64.0510311119170.27915@g5.osdl.org","subject":"Re: git push sends more objects than it needs to","fromName":"Luck, Tony","fromEmail":"tony.luck@intel.com","sentAt":"2005-10-31T19:50:48Z","receivedAt":"2005-10-31T19:50:48Z","isPatch":false,"sender":{"key":"tony.luck@intel.com","avatar":"https://avatars.githubusercontent.com/u/5446021?v=4"},"body":"On Mon, Oct 31, 2005 at 11:36:09AM -0800, Linus Torvalds wrote:\n> a) The \"don't do that then\" approach:\n> \n> Don't go behind git's back and add objects on your own, and expect git \n> to realize what you did. ;^p\n\nTo be a responsible kernel.org git user I need to use packfiles. The\ncheapest and easiest way to do that is to steal them from you.\n\n> b) The \"live with it\" approach:\n> \n> You copied the pack by hand, and that will keep git-unpack-objects \n> from duplicating the objects, but you'll still waste time and network \n> when trying pushing the objects (just once, though).\n> \n> Ie this is what happened this time: nothing really lost, and the end \n> result is fine. Now you know why it happened, and you're fine.\n\nI like this one best. There's a pretty fat pipe between here and\nkernel.org ... more of the time wasted appeared to be on the\nunpacking part than on the transfer.\n\n> d) Just let git do it for you.\n> \n> Copy the pack-files, or add my object directory as an \"alternates\" for \n> your object directory, do the \"git prune-objects\", and then _locally_ \n> on master.kernel.org just do something like\n\nLonger term (when everyone is pulling using the \"git\" protocol)\nadding your object directory to my info/alternates looks to be\nthe best thing.\n\n-Tony\n"}]}