Re: Request for detailed documentation of git pack protocol

From: Jakub Narebski <jnareb@gmail.com>
Date: Jun 3, 2009, 09:21 UTC
Message-ID: <200906031121.38616.jnareb@gmail.com>
In-Reply-To: <20090603012940.GA3355@spearce.org>

On Wed, 3 Jun 2009, Shawn O. Pearce wrote:

Show 36 quoted lines

> Jakub Narebski <jnareb@gmail.com> wrote:
>> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:
>>>>>
>>>>> The client connects and sends the request header. The clone command
>>>>>
>>>>> 	$ git clone git://myserver.com/project.git
>>>>>
>>>>> produces the following request:
>>>>>
>>>>> 	0032git-upload-pack /project.git\\000host=myserver.com\\000
>> [...]
>> 
>> So this mean that when cloning via SSH 
>> 
>>   $ git clone ssh://myserver.com/project.git
>> 
>> instead of this first request git would simply invoke [something like]:
>> 
>>   # ssh myserver.com git-upload-pack project.git
> 
> Actually, 
> 
>     # ssh myserver.com git-upload-pack /project.git
>  
>> isn't it? (I am not sure if it uses "project.git" or "/project.git", 
>> and how it does generate full pathname for repository).
> 
> In an ssh:// format URI, its absolute in the URI, so the / after
> the host name (or port number) is sent as an argument, which is then
> read by the remote git-upload-pack exactly as is, so its effectively
> an absolute path in the remote filesystem.
> 
> In a "user@host:path" format URI, its relative to the user's home
> directory, because we run:
> 
>     # ssh user@host git-upload-pack path

By the way, this accidentally shows why one might want to prefer scp-like / ssh-like "URL" for SSH fetch / push, i.e.

  [user@]myserver.com:/path/to/repo.git/

rather than ssh:// URL version

  ssh://[user@]myserver.com/path/to/repo.git/

On the other hand I think only URL version allows to specify nonstandard port (well, that and ~/.ssh/config).

Show 5 quoted lines

>> BTW I wonder why we use stuffing here using "\0" / NUL as separator
>> trick, and whether line has to be terminated with "\0", or can it be
>> terminated with "\n".
> 
> Stuffing here?  What are we talking about again?

I'm sorry, I was too cryptic here.

I meant that in the request line for fetching via git:// protocol

	0032git-upload-pack /project.git\\000host=myserver.com\\000

you separate path to repository from extra options using "\0" / NUL as a separator. Well, this is only sane separator, as it is path terminator, the only character which cannot appear in pathname (although I do wonder whether project names with e.g. control characters or UTF-8 characters would work correctly).

Is the final terminating character required to be NUL ("\0"), or can it be for LF ("\n"), i.e.

	0032git-upload-pack /project.git\\000host=myserver.com\\n

What options besides (required?) "host=<server>[:<port>]" are supported? Do I understand correctly that "host=<host>" information is required for core.gitProxy to work, isn't it?

Show 8 quoted lines

>>>>  * no-progress
>> 
>> What that does mean?
> 
> The client was started with "git clone -q" or something, and doesn't
> want that side brand 2.  Basically the client just says "I do not
> wish to receive stream 2 on sideband, so do not send it to me,
> and if you did, I will drop it on the floor anyway".

Does this mean that if server does not support "no-progress" capability
then client is required to drop diagnostic by itself? Can client request
to not use sideband (multiplexing) if it is asking for "no-progress";
or is multiplexing required for possible signaling of error condition 
on channel 3?

Show 11 quoted lines

>> It is a bit pity that git protocol was not created with extendability
>> (like capabilities) in mind...
> 
> Yes, no doubt.  There are many things I would have done differently,
> given that I now have 20/20 hindsight vision into the past's future.
> 
> :-)
> 
> The protocol (mostly) works fine as-is.  Its widely distributed in
> terms of clients using it on a daily basis.  Its likely to continue
> to serve our needs well into the future.  So, it is what it is.

I do wonder if existing Internet Standard (in the meaning of RFC) protocols also have such kludges and hacks...

Show 12 quoted lines

>> By the way, how client does know that server started to send final
>> data, i.e. packfile multiplexed / interleaved with progress reports,
>> and should expect <pkt-line-band> rather than <pkt-line> output?
> 
> After the client receives a "ACK" or "NAK" for the number of
> outstanding flushes it still has, *after* it has sent "done".
> This also varies based on whether or not multi_ack was enabled.
> 
> Its ugly.  But basically you keep a running counter of each "flush"
> sent, and then you send a "done" out, and then you wait until
> you have the right number of ACK/NAK answers back, and then the
> stream changes format.

Hmmm... perhaps it would be better if pkt-line-sideband had some distinguishing characteristics from ordinary pkt-line, or that sending multiplexed (with sideband) output was preceded by some signal like "0001" or "0004" or "0005\n", or "000dsideband\n". But as you said hindsight is 20/20.

P.S. By the way, is pkt-line format original invention, or was it 'borrowed' from some other standard or protocol?

-- 
Jakub Narebski
Poland