Re: protocol v2: More data transmitted between client and server since v2.20.0

From: Jeff King <peff@peff.net>
Date: Jan 28, 2019, 21:23 UTC
Message-ID: <20190128212305.GA23587@sigill.intra.peff.net>
In-Reply-To: <CA+ARAtpAN_DJ-zgiwPEBqV1EotgsmggRRQWB59u8O_OPR_kFrw@mail.gmail.com>

On Mon, Jan 21, 2019 at 11:45:20AM +0530, Kaartic Sivaraam wrote:

> I recently came across a blog[1] about how protocol v2 speeds up the
> transfer between the client and the server. It states that the amount
> of data transmitted between the client and the server is less when
> using the protocol v2.

Yes, but...

Show 24 quoted lines

> # Git version: 2.19.0
> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
> git@gitlab.com:gitlab-org/gitlab-ce.git master
> 2>git_protocol_2_git_2.19.0_stderr.txt
> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
> 
> $ wc -l
> 22 git_protocol_2_git_2.19.0_stderr.txt
> 
> 
> # Git version: 2.20.0
> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
> git@gitlab.com:gitlab-org/gitlab-ce.git master
> 2>git_protocol_2_git_2.20.0_stderr.txt
> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
> 04845fdeae75ba5de7c93992a5d55663edf647e0
> refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
> 04845fdeae75ba5de7c93992a5d55663edf647e0
> refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
> d4d4ebadfb373518013382560b1f505eb6217f13
> refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master
> 
> $ wc -l git_protocol_2_git_2.20.0_stderr.txt
> 160971 git_protocol_2_git_2.20.0_stderr.txt

v2.19, etc, were buggy. The rules for ls-remote's pattern matching do not permit us to use ref prefixes (because the documentation specifies that it matches "master" at the end of _any_ ref, not just in the usual spots). But we did anyway. The fix is in 631f0f8c4b (ls-remote: do not send ref prefixes for patterns, 2018-10-31).

That should explain the extra refs, as well (they all match "master", too, and it was wrong that v2.19 did not show them).

It's rather unfortunate, since otherwise ls-remote makes for a nice test of the advertisement code. ;) You can see some difference with "ls-remote --heads", as explained in 6a139cdd74 (ls-remote: pass heads/tags prefixes to transport, 2018-10-31).

The best test is to do a noop git-fetch. E.g.:

  # just make sure we have all the objects. You still see the benefit
  # without it, but if you really want to count bytes, it makes sure
  # you're comparing apples to apples.
  git fetch

  # this has a big useless ref advertisement
  GIT_TRACE_PACKET=$PWD/v0 git -c protocol.version=0 fetch origin master

  # and this is much smaller. Unfortunately it's not just a single line
  # (for "master"), because tag-following requires that the other side
  # tell us about tags, too.
  GIT_TRACE_PACKET=$PWD/v2 git -c protocol.version=2 fetch origin master

  # this one is really tiny
  GIT_TRACE_PACKET=$PWD/v2-notags git -c protocol.version=2 fetch --no-tags origin master

With my origin as https://github.com/gitster/git (which has more heads than the regular git.git), I get:

  $ wc v[02]* | head -n -1
  2441  17098 340884 v0
   733   5828 130263 v2
    22    141   2037 v2-notags

-Peff