[RFC/WIP PATCH 00/11] Protocol version 2, again!

    "Just give us something to play around with" - Peff at GitMerge 2015

This is another approach for updating the pack protocol of Git. While in the previous attempts I tried to come up with the perfect specification of the new protocol I realized that such an approach doesn't lead anywhere. So this time actual working code is included!

The included code is not complete, but a minimal example of

    git config remote.origin.transportversion 2
    git fetch origin master # uses the new protocol!

works.

In a previous discussion[1] towards version 2 of the pack protocol I wanted to come up with a protocol which even includes negotiating what is done in the protocol exchange, such as having a command "push, fetch, ls-remote" being part of the protocol. This is not a very good approach as it's too much work to be done at the same time.

This patch series focusses on just the fetching side, so the first four patches are teaching git-upload-pack about the new pack protocol. The next five patches will teach git fetch and fetch-pack how to use the version 2 protocol. Then there will be a small test and a documentation update.

The new protocol works just like the old protocol, except for the capabilities negotiation being before any exchange of real data.

This solves the problem of having a first huge chunk of data (the refs advertisement) sent over the wire and just realizing in between that we don't need these data for that operation and no way to cancel.

By having a capabilites negotiation first the new protocol may be even more future-proof than the current one, as the capabilites will hopefully be kept small and all larger data transfers will get their own later stage in the protocol.

To determine the protocol version we check the ending of the invoked program for an appended version number to see which protocol version we're using in an exchange. At first I thought we should append a unique suffix instead of a version number to make a distinction easier for the kind of protocol we want to talk (There may be the traditional protocol with no suffix, or the "upload-pack-capabilities-first" protocol which will transmit the capabilities first).

My preference for a string suffix instead of a sequential number is motiviated by the discussion of sha1 vs sequential numbers to describe a state of a repository. The main difference here is however how often we expect changes. Version 1 of the protocol is current for 10 years by now, so I do not changes to the protocol number often, which makes it suitable for just having a counter appended to the binary.

The advantage of just a counting number is its small size, and I think this advantage outweights the disadvantage of having named protocol versions.

This series doesn't include an automatic upgrade of the protocol version for later changes if the server supports it, so for now the use of the new protocol needs to be activated manually via setting remote.origin.transportversion.

Any comments welcome!

Thanks, Stefan

[1] http://permalink.gmane.org/gmane.comp.version-control.git/267572

Nguyễn Thái Ngọc Duy (2):
  upload-pack: make client capability parsing code a separate function
  upload-pack: only accept capabilities on the first "want" line

Stefan Beller (9):
  upload-pack: move capabilities out of send_ref
  upload-pack-2: Implement the version 2 of upload-pack
  transport: add infrastructure to support a protocol version number
  remote.h: add get_remote_capabilities, request_capabilities
  fetch-pack: use the configured transport protocol
  transport: connect_setup appends protocol version number
  transport: get_refs_via_connect exchanges capabilities before refs.
  t5544: add a test case for the new protocol
  Document protocol version 2

 .gitignore                                        |   1 +
 Documentation/technical/pack-protocol.txt         |  86 ++++++++++++--
 Documentation/technical/protocol-capabilities.txt |  15 ---
 Makefile                                          |   2 +
 builtin/fetch-pack.c                              |  17 ++-
 builtin/fetch.c                                   |   6 +
 connect.c                                         |  48 +++++++-
 fetch-pack.h                                      |   1 +
 remote.c                                          |   2 +
 remote.h                                          |   5 +
 t/t5544-fetch-2.sh                                |  40 +++++++
 transport-helper.c                                |   1 +
 transport.c                                       |  60 +++++++++-
 transport.h                                       |   4 +
 upload-pack-2.c                                   |   1 +
 upload-pack.c                                     | 138 +++++++++++++++++-----
 16 files changed, 366 insertions(+), 61 deletions(-)
 create mode 100755 t/t5544-fetch-2.sh
 create mode 120000 upload-pack-2.c