{"thread":{"id":"21961","subject":"the 100 mb push","startedAt":"2009-12-15T19:23:38Z","lastAt":"2009-12-15T20:00:41Z","messageCount":3,"participants":["Joey Hess","Shawn O. Pearce","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"129954","messageId":"20091215192338.GA16654@gnu.kitenet.net","threadId":"21961","inReplyTo":null,"subject":"the 100 mb push","fromName":"Joey Hess","fromEmail":"joey@kitenet.net","sentAt":"2009-12-15T19:23:38Z","receivedAt":"2009-12-15T19:23:38Z","isPatch":false,"sender":{"key":"joey@kitenet.net","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Is it normal for git push to sometimes transfer much more data\nthan seems necessary? Here is a case where that happens:\n\njoey@gnu:~/src/p.t>git branch\n* master\n pristine-tar\n testsuite\njoey@gnu:~/src/p.t>git remote show origin\n* remote origin\n Fetch URL: ssh://joey@git.kitenet.net/srv/git/pristine-tar.test\n Push URL: ssh://joey@git.kitenet.net/srv/git/pristine-tar.test\n HEAD branch: master\n Remote branches:\n master tracked\n pristine-tar tracked\n testsuite tracked\n Local branches configured for 'git pull':\n master merges with remote master\n pristine-tar merges with remote pristine-tar\n testsuite merges with remote testsuite\n Local refs configured for 'git push':\n master pushes to master (fast forwardable)\n pristine-tar pushes to pristine-tar (up to date)\n testsuite pushes to testsuite (local out of date)\n\nHere, master is a typical small project branch. It has a 1 line change\nmade locally.\n\nMeanwhile, the testsuite branch is a 100+ mb monster, containing a lot\nof big binaries. In it, a small change has been made in the origin\nrepo. In the local repo, a *lot* of *big* files have been deleted from\nthe same branch, about 20 mb of files were removed all told. But the diff\nfor this change should be quite small.\n\nSo, testsuite needs to be merged before it can be pushed, but git push\ndoesn't tell me that. Instead, it goes off and does this for 2+ hours:\n\njoey@gnu:~/src/p.t>git push\nCounting objects: 241, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (206/206), done.\nWriting objects: 15% (36/237), 2.16 MiB | 15 KiB/s \n^C\n\nIt seems to be uploading the entire repo over the wire, and this is a\ntypical asymmetric network connection, so that goes slow. (Took me a\nwhile to realize it was not just auto-gcing the repo locally.)\n\nOnce I realized what was going on, it was easy to merge it as shown\nbelow, and then the push transferred an appropriatly small amount of data.\nSo, my question is, assuming this is not a straight up bug in git, would\nit make sense to avoid this gotcha in some way?\n\njoey@gnu:~/src/p.t2>git checkout testsuite\nSwitched to branch 'testsuite'\nYour branch is ahead of 'origin/testsuite' by 1 commit.\njoey@gnu:~/src/p.t2>git pull\nremote: Counting objects: 5, done.\nremote: Compressing objects: 100% (3/3), done.\nremote: Total 3 (delta 2), reused 0 (delta 0)\nUnpacking objects: 100% (3/3), done.\nFrom ssh://git.kitenet.net/srv/git/pristine-tar.test\n 3c16948..fce7ec1 testsuite -> origin/testsuite\nMerge made by recursive.\n Makefile | 5 ++++-\n 1 files changed, 4 insertions(+), 1 deletions(-)\njoey@gnu:~/src/p.t2>git push\nCounting objects: 13, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (8/8), done.\nWriting objects: 100% (8/8), 889 bytes, done.\nTotal 8 (delta 5), reused 0 (delta 0)\nTo ssh://joey@git.kitenet.net/srv/git/pristine-tar.test\n aab45a1..cc93945 master -> master\n fce7ec1..d82f225 testsuite -> testsuite\n\ngit version 1.6.5.3\n\n-- \nsee shy jo\n"},{"id":"129956","messageId":"20091215194212.GC18319@spearce.org","threadId":"21961","inReplyTo":"20091215192338.GA16654@gnu.kitenet.net","subject":"Re: the 100 mb push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-12-15T19:42:12Z","receivedAt":"2009-12-15T19:42:12Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Joey Hess wrote:\n> Is it normal for git push to sometimes transfer much more data\n> than seems necessary? Here is a case where that happens:\n\nYes.\n \n> Meanwhile, the testsuite branch is a 100+ mb monster, containing a lot\n> of big binaries. In it, a small change has been made in the origin\n> repo. In the local repo, a *lot* of *big* files have been deleted from\n> the same branch, about 20 mb of files were removed all told. But the diff\n> for this change should be quite small.\n> \n> So, testsuite needs to be merged before it can be pushed, but git push\n> doesn't tell me that. Instead, it goes off and does this for 2+ hours:\n\nThe problem here is, unlike fetch, push does not do a common\nancestor negotiation. The sending side (your push client)\njust assumes the remote side has *only* what the remote side\nadvertised.\n\nSince the remote side advertised a commit on the testsuite branch\nthat your client doesn't have, your client was forced to assume\nthere was no common ancestor and sent the entire thing.\n\nThis usually doesn't show up that badly because the delta tends to\nbe smaller (no huge binary files), tends to be a strict fast forward\n(so your client contains what the remote advertised), and tags may\nhelp to limit the upload size by being at fixed points in the history\n(so at worst you upload since the last tag).\n\nJunio wrote a patch series for git push over a year ago to make\nit do common ancestor negotiation like git fetch does, but it had\na deadlock problem and the patch series got dropped. Not enough\npeople were interested to help Junio carry it through to being\nready for inclusion.\n \n-- \nShawn.\n"},{"id":"129957","messageId":"7vzl5kypli.fsf@alter.siamese.dyndns.org","threadId":"21961","inReplyTo":"20091215192338.GA16654@gnu.kitenet.net","subject":"Re: the 100 mb push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-12-15T20:00:41Z","receivedAt":"2009-12-15T20:00:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess writes:\n\n> So, my question is, assuming this is not a straight up bug in git, would\n> it make sense to avoid this gotcha in some way?\n\nThe \"push\" support was originally written for people who push into their\nown repositories for publishing (i.e. almost always fast-forwarding) and\nlacked the elaborate common ancestor discovery negotiation the \"fetch\"\nside had.\n\nSuppose you have a rewound or forked history, like this:\n\n\n(their side)\n\n *---Y pu\n / \n *---X\n /\n *---*---Z master\n\n\n(your side)\n\n *---X---*---A pu\n /\n *---*---Z---*---B master\n\n - You were in sync when the 'pu' was at X with them; somebody pushed a\n few commits on top of it (forked case); or\n\n - You were in sync when the 'pu' was at Y with them (you pushed it their\n last time yourself), but you rebuilt 'pu' since then (rewound case).\n\nIf you run \"git push there master +pu\", it learns that the tips of\n'master' and 'pu' are at Z and Y respectively at their end. Because the\nprotocol did not negotiate the common ancestor, it would try to send:\n\n rev-list A B ^Z ^Y\n\nbut using only the information available at your end locally.\n\nBecause you either never have heard of (in a forked case) or no longer\nknow (in a rewound case) what 'Y' is, in order to update 'pu', you end up\nsending commits 'Z..A', duplicating 'Z..X' part, because \"^Y\" cannot\nparticipate in the ancestory computation. Commits 'Z..B' will be sent to\nupdate 'master'.\n\nAnd this aspect of \"git push\" protocol hasn't changed much since it was\nwritten.\n\nIn contrast, the protocol used for \"fetch\" tries to discover 'X' in such a\ncase by a little exchange, like this:\n\n downloader: Your tip is at 'Y'? I've never heard of it. Please\n tell me about its parents.\n\n uploader: It is this \"Y^\", and its parent is 'X', and ...\n\n downloader: Ok, I know what 'X' is. I heard enough to proceed. Thank you.\n"}]}