threads / discuss / 39832

Git Smart HTTP with HTTP/2.0

Subject: Git Smart HTTP with HTTP/2.0

## tl;dr

5 messages between Jul 11, 2015 and Jul 11, 2015.

replies: 4people: 3as markdown or json

ForceCharlie· Jul 11, 2015, 03:10 UTC · lore

As we known, HTTP/2.0 has been released. All Git-Smart-HTTP are currently implemented using HTTP/1.1.

Frequently used Git developers often feel Git HTTP protocol is not satisfactory, slow and unstable.This is because the HTTP protocol itself decides When HTTP/2.0 is published. We might be able to git developers jointly, based on HTTP/2.0 Git-Smart-HTTP service and client support. HTTP/2.0: https://tools.ietf.org/html/rfc7540 Github Mirror: https://httpwg.github.io/specs/rfc7540.html HTTP/2.0 has Flow Controller like SSH, HTTP/2.0 has Server Push POST upload-pack can download large more object-pack

...... libcurl now has begun to support HTTP/2.0, git is also using curl, as well libgit2 may use libcurl I suggest the git of developer joint developer of libgit2 and jgit developers discussion based on HTTP/2.0 Git-Smart-HTTP Now Subversion developer begin write a New HTTP Protocol for svn ("HTTP v2"):http://svn.apache.org/repos/asf/subversion/trunk/notes/http-and-webdav/ http-protocol-v2.txt What about git ?

Ilari Liusvaara· Jul 11, 2015, 07:00 UTC · re: ForceCharlie · lore

Re: Git Smart HTTP with HTTP/2.0

On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote:

> As we known, HTTP/2.0 has been released. All Git-Smart-HTTP are currently
> implemented using HTTP/1.1.

Nit: It is HTTP/2.

> Frequently used Git developers often feel Git HTTP protocol is not
> satisfactory, slow and unstable.This is because the HTTP protocol itself
> decides

Note that there are already two versions of HTTP transport, the old "dumb" one and the newer "smart" one.

The smart one is difficult to speed up (due to nature of the negotiations), but usually is pretty reliable (the efficiency isn't horrible).

Now, the old "dumb" protocol is pretty unreliable and slow. HTTP/2 probably can't do anything with the reliability problems, but probably could improve the speed a bit.

Websockets over HTTP/2 (a.k.a. "websockets2") has not been defined yet. With Websockets(1), it would probably already be possible to tunnel the native git smart transport protocol over it. Probably not worth it.

> When HTTP/2.0 is published. We might be able to git developers jointly,
> based on HTTP/2.0 Git-Smart-HTTP service and client support.
> HTTP/2.0: https://tools.ietf.org/html/rfc7540

Well, it is published already.

-Ilari

Shawn Pearce· Jul 11, 2015, 17:23 UTC · re: Ilari Liusvaara · lore

Re: Git Smart HTTP with HTTP/2.0

On Sat, Jul 11, 2015 at 12:00 AM, Ilari Liusvaara <ilari.liusvaara@elisanet.fi> wrote:

Show 15 quoted lines

> On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote:
>> As we known, HTTP/2.0 has been released. All Git-Smart-HTTP are currently
>> implemented using HTTP/1.1.
>
> Nit: It is HTTP/2.
>
>> Frequently used Git developers often feel Git HTTP protocol is not
>> satisfactory, slow and unstable.This is because the HTTP protocol itself
>> decides
>
> Note that there are already two versions of HTTP transport, the old "dumb"
> one and the newer "smart" one.
>
> The smart one is difficult to speed up (due to nature of the negotiations),
> but usually is pretty reliable (the efficiency isn't horrible).

The negotiation in smart-HTTP actually has some bad corner cases. Each round of negotiation requires a new POST resupplying all previously agreed upon SHA-1s, and a batch of new SHA-1s. We have observed many rounds where this POST is MiBs in size because the peers can't quite agree and have to keep digging through history.

The native protocol on git:// and SSH is not as bad. Negotiation is still many rounds, but these are pipelined and each round does not need to repeat the prior round, as the server has a single stream and is saving state.

Show 7 quoted lines

> Now, the old "dumb" protocol is pretty unreliable and slow. HTTP/2 probably
> can't do anything with the reliability problems, but probably could improve
> the speed a bit.
>
> Websockets over HTTP/2 (a.k.a. "websockets2") has not been defined yet.
> With Websockets(1), it would probably already be possible to tunnel the
> native git smart transport protocol over it. Probably not worth it.

Another option is to tunnel using gRPC (grpc.io). libcurl probably can't do this. And linking grpc.io library into git-core is crazy. So its probably a non-starter. But food for thought.

But, at $DAY_JOB we tunnel the native bidirectional protocol in grpc.io's predecessor, and it works quite well for us.

Ilari Liusvaara· Jul 11, 2015, 18:26 UTC · re: Shawn Pearce · lore

Re: Git Smart HTTP with HTTP/2.0

On Sat, Jul 11, 2015 at 10:23:09AM -0700, Shawn Pearce wrote:

Show 19 quoted lines

> On Sat, Jul 11, 2015 at 12:00 AM, Ilari Liusvaara
> <ilari.liusvaara@elisanet.fi> wrote:
> > On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote:
> >
> >> Frequently used Git developers often feel Git HTTP protocol is not
> >> satisfactory, slow and unstable.This is because the HTTP protocol itself
> >> decides
> >
> > Note that there are already two versions of HTTP transport, the old "dumb"
> > one and the newer "smart" one.
> >
> > The smart one is difficult to speed up (due to nature of the negotiations),
> > but usually is pretty reliable (the efficiency isn't horrible).
> 
> The negotiation in smart-HTTP actually has some bad corner cases. Each
> round of negotiation requires a new POST resupplying all previously
> agreed upon SHA-1s, and a batch of new SHA-1s. We have observed many
> rounds where this POST is MiBs in size because the peers can't quite
> agree and have to keep digging through history.

Oh yeah that... Well, that is artifact of HTTP semantics.

Show 11 quoted lines

> > Now, the old "dumb" protocol is pretty unreliable and slow. HTTP/2 probably
> > can't do anything with the reliability problems, but probably could improve
> > the speed a bit.
> >
> > Websockets over HTTP/2 (a.k.a. "websockets2") has not been defined yet.
> > With Websockets(1), it would probably already be possible to tunnel the
> > native git smart transport protocol over it. Probably not worth it.
> 
> Another option is to tunnel using gRPC (grpc.io). libcurl probably
> can't do this. And linking grpc.io library into git-core is crazy. So
> its probably a non-starter. But food for thought.

Wouldn't it link into git-remote-http (and on the server side, one could use pipes to talk to git)?

But supporting websockets in git-remote-http could get annoying, especially for wss:// (https://). Dunno how bad gRPC would be.

-Ilari

Shawn Pearce· Jul 11, 2015, 23:10 UTC · re: Ilari Liusvaara · lore

Re: Git Smart HTTP with HTTP/2.0

On Sat, Jul 11, 2015 at 11:26 AM, Ilari Liusvaara <ilari.liusvaara@elisanet.fi> wrote:

Show 15 quoted lines

> On Sat, Jul 11, 2015 at 10:23:09AM -0700, Shawn Pearce wrote:
>>
>> > Websockets over HTTP/2 (a.k.a. "websockets2") has not been defined yet.
>> > With Websockets(1), it would probably already be possible to tunnel the
>> > native git smart transport protocol over it. Probably not worth it.
>>
>> Another option is to tunnel using gRPC (grpc.io). libcurl probably
>> can't do this. And linking grpc.io library into git-core is crazy. So
>> its probably a non-starter. But food for thought.
>
> Wouldn't it link into git-remote-http (and on the server side, one
> could use pipes to talk to git)?
>
> But supporting websockets in git-remote-http could get annoying,
> especially for wss:// (https://). Dunno how bad gRPC would be.

We wrote it as git-remote-$THING, invoked with $THING:// URLs. And git-remote-$THING just implements the "connect" helper protocol. Its much simpiler than git-remote-http.

Maybe its done that way in git-core as http2:// aka git-remote-http2.

Or the git-remote-http helper connects to the remote system and tries to guess if it supports Git on HTTP/2 before responding to the capabilities request from transport code. If yes, it replies with connect, if no, it does the current fetch and push protocol.