Re: Request for detailed documentation of git pack protocol

This is combined response to various messages in this thread, following my discoveries done using simple Perl script (using IO::Socket) which assumes role of a git client, tested against github.com (IIRC it uses Ruby implementation) and git.kernel.org (C Git), and "nc -l 9418".

By the way, is there some publicly accessible JGit (Java) and Dulwich (Python) git-daemon one can test against?

  sp = Shawn O. Pearce
  jn = Jakub Narebski
  gb = Git Community Book (http://book.git-scm.com)

jn>> I meant that in the request line for fetching via git:// protocol jn>> jn>>       0032git-upload-pack /project.git\\000host=myserver.com\\000 jn>> jn>> you separate path to repository from extra options using "\0" / NUL jn>> as a separator. Well, this is only sane separator, as it is path jn>> terminator, the only character which cannot appear in pathname jn>> (although I do wonder whether project names with e.g. control jn>> characters or UTF-8 characters would work correctly). sp> sp> No, that isn't the reason '\0' is used here.  But yea, that is true. sp> sp> The reason \0 is used is, git-daemon reads the 4 byte length, decodes sp> that, then reads that many bytes.  Finally it writes a '\0' at the sp> end of what it read, so that the entire "line" is NUL terminated. sp> Then it reads the "command path" part from the resulting C string. sp> sp> The host=myserver.com part came later, after many daemons were sp> already running all over the world.  By hiding it behind the '\0' sp> an old daemon would never see it (but strlen() returned a value that sp> was less than the length read, but the old daemons didn't care). sp> Newer daemons look for where strlen() < length, and assume that sp> the host header follows. sp> sp> The host header ends with '\0' in case additional headers would sp> also appear here in the future.  IOW, like HTTP allows new headers sp> to be added before the "\r\n\r\n" terminator at the body, we allow sp> them between "\0". [...]

sp> The NUL at the end of the host name is not strictly required, but sp> must be present if the client were to ever pass additional options sp> to the server.

Actually both git.kernel.org and github.com failed (deadlocked / hung) when I tried to add extra key=value parameter at the end of request:

  003bgit-upload-pack /project.git\0host=myserver.com\0user=me\0

Hmmmm...

jn>> Hmmm... the communication between server and client is not entirely jn>> clean. Do I understand correctly that this NAK is response to jn>> clients flush after all those "want" lines? sp> sp> Yes. sp> jn>> And that "0009done" from client jn>> tells server that it should send everything it has? sp> sp> Yes.  It means the client will not issue any more "have" lines, sp> as it has nothing further in its history, so the server just has sp> to give up and start generating a pack based on what it knows.

Here we were talking about the following part of exchange: (I have added "C:" prefix to signal that this is what client, git-clone here, sends; I have added also explicit "\n" to mark LF characters terminating lines, and put each pkt-line on separate line)

gb> C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\n gb> C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n gb> C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n gb> C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n gb> C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n gb> C: 0000 gb> C: 0009done\n

and where server response is (again the quote from "Git Community Book" was modified, removing here doublequotes and doubling of backslashes):

gb> S: 0008NAK\n gb> S: 0023\002Counting objects: 2797, done.\n gb> [...] gb> S: 2004\001PACK\000\000\000\002 [...]

I have thought that after sending "0000" flush line client can wait for NAK or ACK server response... but it is not the case. When I tried to read from server after "0000" flush and before "0009done\n", my client (or netcat instance) deadlocked (hung) waiting for server response. I either did a mistake in my fake client, or I don't understand git pack protocol correctly. Should client wait for NAK or ACK from server _only_ after sending maximum number of want/have lines (256 if I remember correctly?)?

When I removed sending "0000" flush line my fake client again hung (deadlocked?) waiting for server.

jn>> P.S. By the way, is pkt-line format original invention, or was it jn>> 'borrowed' from some other standard or protocol? sp> sp> No clue. I find it f'king odd that the length is in hex. There sp> isn't much value to the protocol being human readable. The PACK sp> part of the stream sure as hell ain't. You aren't going to type sp> out a sequence of "have" lines against the remote, like you could sp> with say an HTTP GET. *shrug*

"git gui blame pkt-line.c" shows that pkt-line format is Linus invention.

It looks quite a bit like 'chunked' transfer encoding[1] in HTTP; there each non-empty chunk starts with the number of octets of the data it embeds (size written in hexadecimal) followed by a CRLF (carriage return and linefeed), and the data itself. The chunk is then closed with a CRLF. In some implementations, white space chars (0x20) are padded between chunk-size and the CRLF. In pkt-line format number of octet has fixed width (4 hexadecimal digits, 0-padded), and we do not use CRLF as terminator of chunk/packet length and of chunk/packet itself.

In HTTP 'chunked' transfer encoding the last chunk is a single line, simply made of the chunk-size (0). In pkt-line format we use special size of "0000" for a flush packet.

[1] http://en.wikipedia.org/wiki/Chunked_transfer_encoding