protocol v2: More data transmitted between client and server since v2.20.0

Hi all,

I recently came across a blog[1] about how protocol v2 speeds up the transfer between the client and the server. It states that the amount of data transmitted between the client and the server is less when using the protocol v2.

To demonstrate the difference between v0 and v2 two commands were mentioned in the blog. The results hold true for me when I test using Git v2.18.1. The commands given there worked as mentioned in the blog until Git v2.18.1. But when I run the same commands using Git v2.20.0 or the one I built from source in the 'next' branch (at aa96b0ce6b). I'm not sure why this is the case. Some observations below:

Protocol v0 =========== # Git version: 2.18.1 $ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.18.1_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l git_protocol_0_git_2.18.1_stderr.txt 36848 git_protocol_0_git_2.18.1_stderr.txt

# Git version: 2.19.0 $ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.19.0_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l git_protocol_0_git_2.19.0_stderr.txt 36848 git_protocol_0_git_2.19.0_stderr.txt

# Git version: 2.20.0 $ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.20.0_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l git_protocol_0_git_2.20.0_stderr.txt 36848 git_protocol_0_git_2.20.0_stderr.txt

# Git version: 2.20.1.495.gaa96b0ce6b $ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.20.1-source_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l git_protocol_0_git_2.20.1-source_stderr.txt 36848 git_protocol_0_git_2.20.1-source_stderr.txt

Protocol v2 =========== # Git version: 2.18.1 $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.18.1_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l git_protocol_2_git_2.18.1_stderr.txt 22 git_protocol_2_git_2.18.1_stderr.txt

# Git version: 2.19.0 $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.19.0_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master

$ wc -l 22 git_protocol_2_git_2.19.0_stderr.txt

# Git version: 2.20.0 $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.20.0_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master 04845fdeae75ba5de7c93992a5d55663edf647e0 refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master 04845fdeae75ba5de7c93992a5d55663edf647e0 refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master d4d4ebadfb373518013382560b1f505eb6217f13 refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.0_stderr.txt 160971 git_protocol_2_git_2.20.0_stderr.txt

# Git version: 2.20.1.495.gaa96b0ce6b $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.20.1-source_stderr.txt d4d4ebadfb373518013382560b1f505eb6217f13 refs/heads/master 04845fdeae75ba5de7c93992a5d55663edf647e0 refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master 04845fdeae75ba5de7c93992a5d55663edf647e0 refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master d4d4ebadfb373518013382560b1f505eb6217f13 refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.1-source_stderr.txt 160971 git_protocol_2_git_2.20.1-source_stderr.txt

Also, note that there are more refs printed when using v2 which is spooky too. I thought of running a bisect to find the actual culprit but wanted to know if this was *actually* an issue or not? Any thoughts?

References ========== [1]: https://about.gitlab.com/2018/12/10/git-protocol-v2-enabled-for-ssh-on-gitlab-dot-com/

-- Sivaraam

P.S.: I previously sent this email but it doesn't seemed to have reached the mailing list (as verified by checking the archives) for some unknown reason. So, I've re-sent this email. Sorry if this was redundant.

On Mon, Jan 21, 2019 at 11:45:20AM +0530, Kaartic Sivaraam wrote:

Yes, but...

v2.19, etc, were buggy. The rules for ls-remote's pattern matching do not permit us to use ref prefixes (because the documentation specifies that it matches "master" at the end of _any_ ref, not just in the usual spots). But we did anyway. The fix is in 631f0f8c4b (ls-remote: do not send ref prefixes for patterns, 2018-10-31).

That should explain the extra refs, as well (they all match "master", too, and it was wrong that v2.19 did not show them).

It's rather unfortunate, since otherwise ls-remote makes for a nice test of the advertisement code. ;) You can see some difference with "ls-remote --heads", as explained in 6a139cdd74 (ls-remote: pass heads/tags prefixes to transport, 2018-10-31).

The best test is to do a noop git-fetch. E.g.:

  # just make sure we have all the objects. You still see the benefit
  # without it, but if you really want to count bytes, it makes sure
  # you're comparing apples to apples.
  git fetch

  # this has a big useless ref advertisement
  GIT_TRACE_PACKET=$PWD/v0 git -c protocol.version=0 fetch origin master

  # and this is much smaller. Unfortunately it's not just a single line
  # (for "master"), because tag-following requires that the other side
  # tell us about tags, too.
  GIT_TRACE_PACKET=$PWD/v2 git -c protocol.version=2 fetch origin master

  # this one is really tiny
  GIT_TRACE_PACKET=$PWD/v2-notags git -c protocol.version=2 fetch --no-tags origin master

With my origin as https://github.com/gitster/git (which has more heads than the regular git.git), I get:

  $ wc v[02]* | head -n -1
  2441  17098 340884 v0
   733   5828 130263 v2
    22    141   2037 v2-notags

-Peff

Hi Peff,

Thanks for sharing the details about that spooky behaviour! Also, thanks for suggesting an alternative way to show the advantage of using protocol v2. I'll refer your post in that blog's comment to let them know.

Thanks, Sivaraam