Re: Request for detailed documentation of git pack protocol

Jakub Narebski <jnareb@gmail.com> wrote:

Actually, 

    # ssh myserver.com git-upload-pack /project.git
 

In an ssh:// format URI, its absolute in the URI, so the / after the host name (or port number) is sent as an argument, which is then read by the remote git-upload-pack exactly as is, so its effectively an absolute path in the remote filesystem.

In a "user@host:path" format URI, its relative to the user's home directory, because we run:

    # ssh user@host git-upload-pack path

Stuffing here?  What are we talking about again?
 

Both.  :-)

Clients (old and new) ask for git-upload-pack and git-receive-pack,
by SSH as we saw above... so it needs to be in the remote $PATH.
 

Whatever.  I was talking about what we enumerate, not the output
format.  The server code is actually not using either of those
programs, but is instead just making the direct calls to refs.c
functions (for_each_ref function) and formatting the result specially
for the protocol.
 

It allows the server to return "ACK $SHA1 continue" as soon as it finds a commit that it can use as a common base, between the client's wants and the client's have set.

By sending this early, the server can potentially head off the client from walking any further down that particular branch of the client's repository history. The client may still need to walk down other branches, sending have lines for those, until the server has a complete cut across the DAG, or the client has said "done".

IIRC, without multi_ack, a client sends have lines in --date-order until the server has found a common base. That means the client will send have lines that are already known by the server to be common, because they overlap in time with another branch that the server hasn't found a common base on yet.

E.g. the client has things in caps that the server doesn't; server has things in lower case:

     +---- u ---------------------- x
    /             +----- y
   /             /
  a -- b -- c -- d -- E -- F
   \
    +--- Q -- R -- S

If the client wants x,y and starts out by saying have F,S, the server doesn't know what F,S is. Eventually the client says "have d" and the server sends "ACK d continue" to let the client know to stop walking down that line (so don't send c-b-a), but its not done yet, it needs a base for X. The client keeps going with S-R-Q, until a gets reached, at which point the server has a clear base and it all ends.

Without multi_ack the client would have sent that c-b-a chain anyway, interleaved with S-R-Q.

Junio, am I right? I think I am, but I've had to reverse engineer most of this. And the above is my understanding of it.

Yes.
 

No, no it doesn't. But if the client does request it (and I think modern clients actually do request it, even on initial clone case) the server won't produce a thin pack. Why? There is no common base, so there is no uninteresting set to omit from the pack. :-)

Yes. These two options are mutually exclusive. A client should ask for only one of them, and a modern client always favors side-band-64k.

Long ago, we only had side-band, which allowed up to 1000 bytes per packet. But the packet length field is 4 bytes, in hex, so 16 bits worth of information space. Limiting it to only 1000 bytes for a large 800 MiB binary pack file on initial clone is really quite poor usage of the data stream space.

We couldn't just up the limit the server sends to the full 2^16
because older clients literally had a char[1000] allocated on the
stack, and we'd overflow it.  So "side-band-64k" came about as
another way for the client to request side-band, but to also say
it can handle the much larger packets, packets that are actually
crammed nearly full (65520 bytes).
 

Yes.  Its that they can send/read OBJ_OFS_DELTA, aka type 6 in
a pack file.
 

The client was started with "git clone -q" or something, and doesn't
want that side brand 2.  Basically the client just says "I do not
wish to receive stream 2 on sideband, so do not send it to me,
and if you did, I will drop it on the floor anyway".
 

It doesn't, but convention says to include the "\n" to be nice to a human. Junio may argue that it does require it, I don't know, but if you read through any modern implementation (e.g. C Git or JGit) the "\n" is entirely optional when parsing the lines.

Actually, it has to be, because that "\n" isn't there on the first
line when the capability data appears either wedged behind the "\0"
or after the " " at the end of the line.
 

Yes, no doubt. There are many things I would have done differently, given that I now have 20/20 hindsight vision into the past's future.

:-)

The protocol (mostly) works fine as-is.  Its widely distributed in
terms of clients using it on a daily basis.  Its likely to continue
to serve our needs well into the future.  So, it is what it is.
 

Invalid. No clue if they are considered "reserved for future use". I don't think they are, I think they just out flat out not something any client can ever sanely produce.

But, hey, look, another back door we can use in the future to
wedge something else into this protocol, after introducing a new
capability for it.  :-)
 

Yes.

Yes. It means the client will not issue any more "have" lines, as it has nothing further in its history, so the server just has to give up and start generating a pack based on what it knows.

Correct.
 

After the client receives a "ACK" or "NAK" for the number of outstanding flushes it still has, *after* it has sent "done". This also varies based on whether or not multi_ack was enabled.

Its ugly.  But basically you keep a running counter of each "flush"
sent, and then you send a "done" out, and then you wait until
you have the right number of ACK/NAK answers back, and then the
stream changes format.