{"thread":{"id":"50270","subject":"protocol v2: More data transmitted between client and server since v2.20.0","startedAt":"2019-01-21T07:20:15Z","lastAt":"2019-02-02T18:01:36Z","messageCount":3,"participants":["Kaartic Sivaraam","Jeff King","Kaartic.Sivaraam"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"367215","messageId":"CA+ARAtpAN_DJ-zgiwPEBqV1EotgsmggRRQWB59u8O_OPR_kFrw@mail.gmail.com","threadId":"50270","inReplyTo":null,"subject":"protocol v2: More data transmitted between client and server since v2.20.0","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2019-01-21T06:15:20Z","receivedAt":"2019-01-21T07:20:15Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Hi all,\n\nI recently came across a blog[1] about how protocol v2 speeds up the\ntransfer between the client and the server. It states that the amount\nof data transmitted between the client and the server is less when\nusing the protocol v2.\n\nTo demonstrate the difference between v0 and v2 two commands were\nmentioned in the blog. The results hold true for me when I test using\nGit v2.18.1. The commands given there worked as mentioned in the blog\nuntil Git v2.18.1. But when I run the same commands using Git v2.20.0\nor the one I built from source in the 'next' branch (at aa96b0ce6b).\nI'm not sure why this is the case. Some observations below:\n\nProtocol v0\n===========\n# Git version: 2.18.1\n$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_0_git_2.18.1_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l git_protocol_0_git_2.18.1_stderr.txt\n36848 git_protocol_0_git_2.18.1_stderr.txt\n\n\n# Git version: 2.19.0\n$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_0_git_2.19.0_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l git_protocol_0_git_2.19.0_stderr.txt\n36848 git_protocol_0_git_2.19.0_stderr.txt\n\n\n# Git version: 2.20.0\n$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_0_git_2.20.0_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l git_protocol_0_git_2.20.0_stderr.txt\n36848 git_protocol_0_git_2.20.0_stderr.txt\n\n\n# Git version: 2.20.1.495.gaa96b0ce6b\n$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_0_git_2.20.1-source_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l git_protocol_0_git_2.20.1-source_stderr.txt\n36848 git_protocol_0_git_2.20.1-source_stderr.txt\n\n\nProtocol v2\n===========\n# Git version: 2.18.1\n$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_2_git_2.18.1_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l git_protocol_2_git_2.18.1_stderr.txt\n22 git_protocol_2_git_2.18.1_stderr.txt\n\n\n# Git version: 2.19.0\n$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_2_git_2.19.0_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n\n$ wc -l\n22 git_protocol_2_git_2.19.0_stderr.txt\n\n\n# Git version: 2.20.0\n$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_2_git_2.20.0_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n04845fdeae75ba5de7c93992a5d55663edf647e0\nrefs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master\n04845fdeae75ba5de7c93992a5d55663edf647e0\nrefs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master\nd4d4ebadfb373518013382560b1f505eb6217f13\nrefs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master\n\n$ wc -l git_protocol_2_git_2.20.0_stderr.txt\n160971 git_protocol_2_git_2.20.0_stderr.txt\n\n\n# Git version: 2.20.1.495.gaa96b0ce6b\n$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\ngit@gitlab.com:gitlab-org/gitlab-ce.git master\n2>git_protocol_2_git_2.20.1-source_stderr.txt\nd4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n04845fdeae75ba5de7c93992a5d55663edf647e0\nrefs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master\n04845fdeae75ba5de7c93992a5d55663edf647e0\nrefs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master\nd4d4ebadfb373518013382560b1f505eb6217f13\nrefs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master\n\n$ wc -l git_protocol_2_git_2.20.1-source_stderr.txt\n160971 git_protocol_2_git_2.20.1-source_stderr.txt\n\n\nAlso, note that there are more refs printed when using v2 which is\nspooky too. I thought of running a bisect to find the actual culprit\nbut wanted to know if this was *actually* an issue or not? Any\nthoughts?\n\n\nReferences\n==========\n[1]: https://about.gitlab.com/2018/12/10/git-protocol-v2-enabled-for-ssh-on-gitlab-dot-com/\n\n--\nSivaraam\n\nP.S.: I previously sent this email but it doesn't seemed to have reached the\nmailing list (as verified by checking the archives) for some unknown reason.\nSo, I've re-sent this email. Sorry if this was redundant.\n"},{"id":"367866","messageId":"20190128212305.GA23587@sigill.intra.peff.net","threadId":"50270","inReplyTo":"CA+ARAtpAN_DJ-zgiwPEBqV1EotgsmggRRQWB59u8O_OPR_kFrw@mail.gmail.com","subject":"Re: protocol v2: More data transmitted between client and server since v2.20.0","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-01-28T21:23:05Z","receivedAt":"2019-01-28T21:23:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 21, 2019 at 11:45:20AM +0530, Kaartic Sivaraam wrote:\n\n> I recently came across a blog[1] about how protocol v2 speeds up the\n> transfer between the client and the server. It states that the amount\n> of data transmitted between the client and the server is less when\n> using the protocol v2.\n\nYes, but...\n\n> # Git version: 2.19.0\n> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\n> git@gitlab.com:gitlab-org/gitlab-ce.git master\n> 2>git_protocol_2_git_2.19.0_stderr.txt\n> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n> \n> $ wc -l\n> 22 git_protocol_2_git_2.19.0_stderr.txt\n> \n> \n> # Git version: 2.20.0\n> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote\n> git@gitlab.com:gitlab-org/gitlab-ce.git master\n> 2>git_protocol_2_git_2.20.0_stderr.txt\n> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master\n> 04845fdeae75ba5de7c93992a5d55663edf647e0\n> refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master\n> 04845fdeae75ba5de7c93992a5d55663edf647e0\n> refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master\n> d4d4ebadfb373518013382560b1f505eb6217f13\n> refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master\n> \n> $ wc -l git_protocol_2_git_2.20.0_stderr.txt\n> 160971 git_protocol_2_git_2.20.0_stderr.txt\n\nv2.19, etc, were buggy. The rules for ls-remote's pattern matching do\nnot permit us to use ref prefixes (because the documentation specifies\nthat it matches \"master\" at the end of _any_ ref, not just in the usual\nspots). But we did anyway. The fix is in 631f0f8c4b (ls-remote: do not\nsend ref prefixes for patterns, 2018-10-31).\n\nThat should explain the extra refs, as well (they all match \"master\",\ntoo, and it was wrong that v2.19 did not show them).\n\nIt's rather unfortunate, since otherwise ls-remote makes for a nice test\nof the advertisement code. ;) You can see some difference with\n\"ls-remote --heads\", as explained in 6a139cdd74 (ls-remote: pass\nheads/tags prefixes to transport, 2018-10-31).\n\nThe best test is to do a noop git-fetch. E.g.:\n\n  # just make sure we have all the objects. You still see the benefit\n  # without it, but if you really want to count bytes, it makes sure\n  # you're comparing apples to apples.\n  git fetch\n\n  # this has a big useless ref advertisement\n  GIT_TRACE_PACKET=$PWD/v0 git -c protocol.version=0 fetch origin master\n\n  # and this is much smaller. Unfortunately it's not just a single line\n  # (for \"master\"), because tag-following requires that the other side\n  # tell us about tags, too.\n  GIT_TRACE_PACKET=$PWD/v2 git -c protocol.version=2 fetch origin master\n\n  # this one is really tiny\n  GIT_TRACE_PACKET=$PWD/v2-notags git -c protocol.version=2 fetch --no-tags origin master\n\nWith my origin as https://github.com/gitster/git (which has more heads\nthan the regular git.git), I get:\n\n  $ wc v[02]* | head -n -1\n  2441  17098 340884 v0\n   733   5828 130263 v2\n    22    141   2037 v2-notags\n\n-Peff\n"},{"id":"368419","messageId":"0da174c7-40cd-c245-abbf-6aa7a657f5a3@gmail.com","threadId":"50270","inReplyTo":"20190128212305.GA23587@sigill.intra.peff.net","subject":"Re: protocol v2: More data transmitted between client and server since v2.20.0","fromName":"Kaartic.Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2019-02-02T18:01:26Z","receivedAt":"2019-02-02T18:01:36Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Hi Peff,\n\nThanks for sharing the details about that spooky behaviour! Also, thanks \nfor suggesting an alternative way to show the advantage of using \nprotocol v2. I'll refer your post in that blog's comment to let them know.\n\n\nThanks,\nSivaraam\n"}]}