{"thread":{"id":"39165","subject":"Question about how git determines the minimum packfile for a push.","startedAt":"2015-04-27T00:41:28Z","lastAt":"2015-04-28T05:33:32Z","messageCount":3,"participants":["Brad Litterell","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"260046","messageId":"E51394554A503C4E852F9BEE46B03E8D01E4E784@TI-ODIN.tasernet.com","threadId":"39165","inReplyTo":null,"subject":"Question about how git determines the minimum packfile for a push.","fromName":"Brad Litterell","fromEmail":"brad@evidence.com","sentAt":"2015-04-27T00:41:28Z","receivedAt":"2015-04-27T00:41:28Z","isPatch":false,"sender":{"key":"brad@evidence.com","avatar":null},"body":"Hi,\n\nI'm using git with a submodule containing a (large) binary toolchain where I updated the version from GCC-4.7 to 4.8. When I added 4.8 I deleted 4.7 and now want to add 4.7 back to the HEAD. As shown in the tree objects below, the 4.7 bits are still in the repository (as expected), but when I try to push a commit and tree object that puts them back, git tries to send all 4000+ objects again, even though the objects are already be on the server from a previous commit. (I have confirmed this with git cat-file on the server.) From my research it seems like git is supposed to identify the common objects and not do this.\n\nFor example, consider the last three commits (most recent on top).\n\n$ git log\nbb384915e12e925ead5ab8ad5161c84e0ef2b7f7 (HEAD, master) Add GCC-4.7 back to the image for side-by-side testing. Will delete it later.\n2dfd226e6d2cc0a1dc58770d1dcaec1ba863df72 (origin/master) Upgrade toolchain to GCC-4.8. (delete GCC-4.7)\n816fde0fdec1506600f19de4e3e4e02a6fe08639 (tag: release-1) Compiler toolchain 4.7\n\nHere are the internal tree objects for those 3 commits:\n\n# git cat-file -p 816fde^{tree}\n$ cat old.tree\n100644 blob 841b8359fce4edaf7549b87e1a81e7091c1dff6c .arcconfig\n100644 blob 21cb788d9f6e99367d96ba19d8c7470164e7d298 .gitmodules\n040000 tree 470dfed1791b3ab7b4c731e354fd7685609c057a arcanist\n100644 blob 588866ae2c8a815ff26c5930b98d2a8ac9b934a0 gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux.tar.bz2\n040000 tree 7a1ae67dd2ff1fd358ad52d25852b340355339fe gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux\n\n# git cat-file -p 2dfd22^{tree}\n$ cat middle.tree\n100644 blob 841b8359fce4edaf7549b87e1a81e7091c1dff6c .arcconfig\n100644 blob 21cb788d9f6e99367d96ba19d8c7470164e7d298 .gitmodules\n100644 blob 79c4d3271c1177822786f82201c80b928fc35c6e README\n040000 tree 470dfed1791b3ab7b4c731e354fd7685609c057a arcanist\n100644 blob f71d6dd4e13a3de4b0c38dd37e4c2bc94c503f26 gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux.tar.bz2\n100644 blob 026e2d232bd7bb1cf0b9efb61c8cd307a52526ec gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux.tar.bz2.asc\n040000 tree d03b4cf20163ca7a0ab5c02365890855146d8e0c gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux\n\n# git cat-file -p HEAD^{tree}\n$ cat new.tree\n100644 blob 841b8359fce4edaf7549b87e1a81e7091c1dff6c .arcconfig\n100644 blob 21cb788d9f6e99367d96ba19d8c7470164e7d298 .gitmodules\n100644 blob 79c4d3271c1177822786f82201c80b928fc35c6e README\n040000 tree 470dfed1791b3ab7b4c731e354fd7685609c057a arcanist\n100644 blob 588866ae2c8a815ff26c5930b98d2a8ac9b934a0 gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux.tar.bz2\n040000 tree 7a1ae67dd2ff1fd358ad52d25852b340355339fe gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux\n100644 blob f71d6dd4e13a3de4b0c38dd37e4c2bc94c503f26 gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux.tar.bz2\n100644 blob 026e2d232bd7bb1cf0b9efb61c8cd307a52526ec gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux.tar.bz2.asc\n040000 tree d03b4cf20163ca7a0ab5c02365890855146d8e0c gcc-linaro-arm-linux-gnueabihf-4.8-2013.12_linux\n\nBy examining the tree objects with git cat-file -p it is clear that adding back GCC-4.7 added back the same exact blobs, which was expected, namely:\n\n100644 blob 588866ae2c8a815ff26c5930b98d2a8ac9b934a0 gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux.tar.bz2\n040000 tree 7a1ae67dd2ff1fd358ad52d25852b340355339fe gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux\n\nBut, whenever I try to push, git tries to write 4000+ objects across the wire:\n\n$ git push origin master\nCounting objects: 4854, done.\nDelta compression using up to 6 threads.\nCompressing objects: 100% (2295/2295), done.\nWriting objects: 0% (2/4853), 9.26 MiB | 3.08 MiB/s\n^C \n\nAll the parts are already on the server so it seems like the only objects to upload would be the commit object and the new associated tree (not the blob (5888) and tree (7a1a) below it).\n\nFurther, it seems like my local git client should be able to discern this from its knowledge of the origin/master ref.\n\nI don't want to let this push complete if it will result in duplication on the server because this repo is already 400MB and slow to clone.\n\nCan someone please explain what is happening here? using git push -thin doesn't seem to make a difference.\n\nIs it possible git is not computing the delta correctly? Or does git only look at the top-level commit objects to figure out what to include in the push packfile?\n\nWill it upload the larger pack only to have the server correctly handle the duplicates?\n\nIn case it matters, the server in question is running Atlassian Stash.\n\nThanks,\nBrad\n"},{"id":"260047","messageId":"CAPc5daV2GmtaknNhrBc7ofMZPsKPUn81SHBVPNdHHTXJs8ubdQ@mail.gmail.com","threadId":"39165","inReplyTo":"E51394554A503C4E852F9BEE46B03E8D01E4E784@TI-ODIN.tasernet.com","subject":"Re: Question about how git determines the minimum packfile for a push.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-27T04:39:32Z","receivedAt":"2015-04-27T04:39:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Sun, Apr 26, 2015 at 5:41 PM, Brad Litterell wrote:\n>\n> Is it possible git is not computing the delta correctly?\n> Or does git only look at the top-level commit objects to figure out what to\n> include in the push packfile?\n\nWe walk the commit graph backwards to discover the common ancestries to\nminimize the network cost when fetching, but I do not think the\nreverse direction\nhas such smart in the protocol.\n\nIf you fetch (not \"pull\") first to remote tracking branches and then push, that\nprobably will reduce the transfer, as the side that pushes is the only one that\ndecides what objects are sent in \"git push -> git receive-pack\" direction.\n"},{"id":"260099","messageId":"20150428053332.GH24580@peff.net","threadId":"39165","inReplyTo":"E51394554A503C4E852F9BEE46B03E8D01E4E784@TI-ODIN.tasernet.com","subject":"Re: Question about how git determines the minimum packfile for a push.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-28T05:33:32Z","receivedAt":"2015-04-28T05:33:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 27, 2015 at 12:41:28AM +0000, Brad Litterell wrote:\n\n> Is it possible git is not computing the delta correctly? Or does git\n> only look at the top-level commit objects to figure out what to\n> include in the push packfile?\n\nIt's the latter. Junio mentioned that \"push\" is not as thorough about\nfinding common ancestors as \"fetch\", but I think even \"fetch\" would have\nthe same problem.\n\nIf we know that the other side has commit X, we know that it also has\nX~3, and we also know that it has every tree and blob mentioned by X~3.\nBut it's much too expensive to open up every tree to generate the full\nset of reachable objects; for the Linux kernel, that is something like 45\nseconds of CPU time, just to find out \"oh, we only need to send 5\nobjects\".\n\nThis works pretty well in practice, because trees and blobs from older\nhistory don't tend to resurface verbatim. But as you noticed, there are\ncertain cases where it does happen, and the number of objects affected\ncan be quite large (to the point that sending the extra objects is much\nmore expensive than the cost of doing the extra tree traversal).\nUnfortunately there is no \"look harder\" option you can give to\n\"git push\" when you, as the user, realize this is happening.\n\nIf you have pack reachability bitmaps, they do produce a more thorough\nanswer. So probably:\n\n git repack -adb\n git push\n\non the client would make this work as you expect.\n\n> Will it upload the larger pack only to have the server correctly handle the duplicates?\n\nYes, the receiving side should correctly handle the duplicates.\n\n-Peff\n"}]}