{"thread":{"id":"2159","subject":"Revamping the git protocol","startedAt":"2005-10-20T04:31:29Z","lastAt":"2005-10-21T01:04:29Z","messageCount":10,"participants":["H. Peter Anvin","Junio C Hamano","Petr Baudis","Linus Torvalds","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10336","messageId":"43571DA1.6030907@zytor.com","threadId":"2159","inReplyTo":null,"subject":"Revamping the git protocol","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-20T04:31:29Z","receivedAt":"2005-10-20T04:31:29Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Okay, so I've started thinking about what it would take to revamp the \ngit protocol.  What I came up with seems a little complex, but all it \nreally is is take the framework that most successful Internet protocols \nhave used and applied it to git.\n\nSomething else that I've noticed is that there is functionality overlap \nbetween git-daemon and git-send-pack, such as the namespace management \n(DWIM functionality.)  Additionally, even when using git over ssh there \nis the potential for version skew, so it might be worthwhile to run the \nfull protocol over ssh as well.\n\nAnyway, here is a strawman.  Items I feel unsure about I've put in brackets.\n\n  ----------\n\n1. \"Strings\" are sequences of bytes prefixed with a length.  The length \nis encoded as four lower-case hexadecimal digits.  [Why not as 2 or 4 \nbytes of network byte order binary?]  When represented in this text as \n\"foo\", this means the sequence of bytes on the wire is <0003foo>.\n\n2. Upon connection, the server will issue a sequence of strings, \nterminated by a null string.  The first string will be of the format:\n\n\"git <x.y>[ <hostname>]\"\n\nx.y is protocol revision (currently 1.0) with the following semantics:\n\n- a change in x indicates a fully incompatible protocol change which \nmeans a client which doesn't understand the exact x version should \nimmediately disconnect without issuing any output.\n\n- a change in y indicates a backward-compatible protocol change which \nmenas a client which understands an older version of the protocol can \nstill communicate.\n\n- hostname is an optional canonical name for this server.\n\nFor protocol version 1.0, subsequent strings are of the form:\n\n\"<R|O|I> option[ <parameters...>]\"\n\n... where the letter indicates REQUIRED, OPTIONAL or INFORMATIVE.  If a \nserver specifies a REQUIRED option which the client does not understand \nor support, the CLIENT should terminate with an \"unable\" command (see \nbelow).  An OPTIONAL option is available to the client should it choose \nto accept it.  An INFORMATIVE option has no protocol function, but may \nbe used to tune the client, inform the client of server policies (such \nas timeouts) or display to the end user if the client is in verbose mode.\n\nNote that the addition of options does not require a new protocol \nrevision.  It is generally believed that the protocol revision will \nrarely, if ever, be changed.\n\n2a. Option \"challenge\":\n\n     \"R challenge <seed>\"\n\n... where 'seed' is any sequence of bytes means that the client should \ncompute the SHA-1 of the seed and issue a \"response\" command with the \nSHA1 in hexadecimal form before issuing any other command.\n\n3. After receiving the list of options, the client can issue commands. \nCommands are strings beginning with a command, one space, and any \narguments as appropriate to the command.\n\n4. The response to a command is a string beginning with a dot-separated \nsequence of numbers, one space, and an optional human-readable text \nstring.  Each part of the dot-separated sequence refines the response; \nif a client receives \"3.1.1.6 foo\" and doesn't know what it is, but \nknows what a \"3.1\" response is, it should treat the 3.1.1.6 response as \na 3.1 response.\n\nIf the server is closing the connection, the response is prefixed with \nthe letter 'C':\n\n\"C5.0.1 Incorrect response\"\n\nFuture versions of the protocol might define new prefix letters; if a \nclient encounters unknown prefix letters they should be ignored.\n\n2\t- successful completion, closing connection\n3\t- successful initiation, begin transaction\n4\t- transient error\n4.1\t- server resource exhaustion errors\n4.1.1\t- load too high\n5\t- permanent error\n5.1\t- protocol errors\n5.2\t- authentication error\n5.2.1\t- invalid reponse to challenge option\n5.3\t- permission errors\n5.3.1\t- repository access denied\n5.4\t- data integrity error\n5.4.1\t- invalid or corrupt repository\n\n5. Commands, and their responses:\n\n\"response <sha1>\"\n\n... response to a \"challenge\" option.  Responses:\n\n\"2.0 OK\" - response accepted\n\"C5.2.1 Invalid response\" - invalid response\n\n\"unable <human error message>\"\n\n... error message from the client to the server due to an unsupported R \noption.  Sending this message can inform the server administrator of \nversion skew problems.\n\nResponse:\n\n\"C5.1.1 Too bad\"\n\n\"send-pack <path>\"\n\n... begin synchronization of the repository at <path>.  Responses:\n\n\"3.1.1 Begin\"\nAny 4.1 response\nAny 5.3 or 5.4 response\n\n\nClearly this needs to be fleshed out a bit more... is this total \ninsanity on my part, or is this something worth doing?\n\n\t-hpa\n"},{"id":"10343","messageId":"7vwtk8pvju.fsf@assigned-by-dhcp.cox.net","threadId":"2159","inReplyTo":"43571DA1.6030907@zytor.com","subject":"Re: Revamping the git protocol","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-20T06:11:17Z","receivedAt":"2005-10-20T06:11:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Wow.\n\n\n\n\n\n\n\n\n\n\n\nThat's elaborate.  And all this is to replace the beginning of\nexecute() part of daemon.c?  What I am assuming is that after\nexchanging command-response initially, you still plan to\neventually have the protocol driver such as upload-pack to take\nthings over, once \"send-pack <path>\" is issued, but is my\nassumption correct?  Or are you also thinking about redoing\nupload-pack as well (otherwise you cannot issue 5.4 errors)?\n\nI am wondering if we can just get away with a simpler scheme\nLinus outlined instead.  One drawback of that approach is it\ndoes not easily allow things like challenge-response uniformly\nacross different commands (admittedly we only have \"upload-pack\"\ncommand right now, but we could add list of supported commands\neasily in execute()), but you could do something along this, I\npresume?\n\nWhen daemon is started with --require-challenge-response,\nthe client needs to issue \"challenge-me\" command and complete\nchallenge_response successfully before being able to issue any\nother commands.\n\nNOTE: this is just an outline, not a compilable patch.  You need to\nfill in the details of challenge response, definition of\n\"require_challenge_response\" variable of type bool, and a\ncommand line parsing to set that variable.\n\n\n---\n\ngit diff\ndiff --git a/daemon.c b/daemon.c\nindex c3381b3..8a8746a 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -204,20 +204,55 @@ static int upload(char *dir)\n \treturn -1;\n }\n \n-static int execute(void)\n+static int challenge_response(const char *me)\n {\n-\tstatic char line[1000];\n-\tint len;\n+\tchar line[1000];\n \n-\talarm(init_timeout ? init_timeout : timeout);\n+\tpacket_write(1, \"here comes your challenge\");\n+\n+\talarm(timeout);\n \tlen = packet_read_line(0, line, sizeof(line));\n \talarm(0);\n \n \tif (len && line[len-1] == '\\n')\n \t\tline[--len] = 0;\n \n-\tif (!strncmp(\"git-upload-pack /\", line, 17))\n-\t\treturn upload(line+16);\n+\tif (\"validate response we obtained in line here\")\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int execute(void)\n+{\n+\tstatic char line[1000];\n+\tint len;\n+\tint client_ok = !require_challenge_response;\n+\tunsigned int time_out = init_timeout;\n+\n+\twhile (1) {\n+\n+\t\talarm(time_out);\n+\t\ttime_out = timeout;\n+\t\tlen = packet_read_line(0, line, sizeof(line));\n+\t\talarm(0);\n+\t\tif (len && line[len-1] == '\\n')\n+\t\t\tline[--len] = 0;\n+\n+\t\tif (!strncmp(\"challenge-me \", line, 13)) {\n+\t\t\tclient_ok = challenge_response(line+13);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!client_ok)\n+\t\t\tbreak;\n+\n+\t\tif (!strncmp(\"git-upload-pack /\", line, 17))\n+\t\t\treturn upload(line+16);\n+\n+\t\t/* more commands here later */\n+\n+\t\tbreak;\n+\t}\n \n \tlogerror(\"Protocol error: '%s'\", line);\n \treturn -1;\n"},{"id":"10349","messageId":"20051020091245.GY30889@pasky.or.cz","threadId":"2159","inReplyTo":"7vwtk8pvju.fsf@assigned-by-dhcp.cox.net","subject":"Re: Revamping the git protocol","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-10-20T09:12:45Z","receivedAt":"2005-10-20T09:12:45Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Thu, Oct 20, 2005 at 08:11:17AM CEST, I got a letter\nwhere Junio C Hamano <junkio@cox.net> told me that...\n> I am wondering if we can just get away with a simpler scheme\n> Linus outlined instead.  One drawback of that approach is it\n> does not easily allow things like challenge-response uniformly\n> across different commands (admittedly we only have \"upload-pack\"\n> command right now, but we could add list of supported commands\n> easily in execute()), but you could do something along this, I\n> presume?\n\nWhat's wrong with my scheme? That is, _reply_ with challenge to the\nupload-pack command. This should be equally powerful to the Linus'\nscheme and the crucial advantage is that you do not need to tell at\nthe client side whether you are talking to a new server or an old one.\n\nI was convinced that the authentication part of the challenge-resposne\nisn't such a good idea after all, though. ;-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"10358","messageId":"4357BCBC.1020706@zytor.com","threadId":"2159","inReplyTo":"20051020091245.GY30889@pasky.or.cz","subject":"Re: Revamping the git protocol","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-20T15:50:20Z","receivedAt":"2005-10-20T15:50:20Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Petr Baudis wrote:\n> Dear diary, on Thu, Oct 20, 2005 at 08:11:17AM CEST, I got a letter\n> where Junio C Hamano <junkio@cox.net> told me that...\n> \n>>I am wondering if we can just get away with a simpler scheme\n>>Linus outlined instead.  One drawback of that approach is it\n>>does not easily allow things like challenge-response uniformly\n>>across different commands (admittedly we only have \"upload-pack\"\n>>command right now, but we could add list of supported commands\n>>easily in execute()), but you could do something along this, I\n>>presume?\n> \n> What's wrong with my scheme? That is, _reply_ with challenge to the\n> upload-pack command. This should be equally powerful to the Linus'\n> scheme and the crucial advantage is that you do not need to tell at\n> the client side whether you are talking to a new server or an old one.\n> \n> I was convinced that the authentication part of the challenge-resposne\n> isn't such a good idea after all, though. ;-)\n> \n\nAnyone noticed that either of those schemes aren't actually \nbackward-compatible in any way (old client talking to new server will be \ndisconnected), and that unfortunately is the best thing one can do with \nthe current setup, exactly because there is no option negotiation phase?\n\nAnother issue is that currently there is no error information propagated \nback to the client; the server logs an error in its own logs, but the \nclient is simply disconnected.\n\n\t-hpa\n"},{"id":"10359","messageId":"Pine.LNX.4.64.0510200910160.3369@g5.osdl.org","threadId":"2159","inReplyTo":"43571DA1.6030907@zytor.com","subject":"Re: Revamping the git protocol","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-20T16:20:09Z","receivedAt":"2005-10-20T16:20:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 19 Oct 2005, H. Peter Anvin wrote:\n> \n> 1. \"Strings\" are sequences of bytes prefixed with a length.  The length is\n> encoded as four lower-case hexadecimal digits.  [Why not as 2 or 4 bytes of\n> network byte order binary?]  When represented in this text as \"foo\", this\n> means the sequence of bytes on the wire is <0003foo>.\n\nAs a reason for your \"why\" - imagine debugging a protocol using telnet..\n\nASCII really is very nice for things like that.\n\nAnd no, \"foo\" is not represented as <0003foo>. It's represented as \n<0007foo>, because the length includes the length of the prefix.\n\nThe special sequence <0000> is a flush sequence, and it's designed so that \nit's supposed to be distinguishable from an empty string <0004>. A <0001> \nto <0003> will be rejected as an error. Maximum string length is thus \n65531.\n\n(Actually, right now flush it is _not_ distinguishable from an empty \nstring because we return 0 for both cases from packet_read_line(), but the \npoint being that the packet protocol _supports_ it being distinguishable \nif we ever need it to).\n\n\t\t\tLinus\n"},{"id":"10360","messageId":"Pine.LNX.4.64.0510200924110.3369@g5.osdl.org","threadId":"2159","inReplyTo":"20051020091245.GY30889@pasky.or.cz","subject":"Re: Revamping the git protocol","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-20T16:38:08Z","receivedAt":"2005-10-20T16:38:08Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 20 Oct 2005, Petr Baudis wrote:\n> \n> What's wrong with my scheme? That is, _reply_ with challenge to the\n> upload-pack command.\n\nNeither your not Peter's scheme seems to be at all worried about backwards \ncompatibility, and I just don't see _why_.\n\nEven if you can upgrade all servers (there aren't that many of them), why \nforce a client upgrade when the protocol is designed to be extensible?\n\nEspecially for somethign that doesn't even _buy_ you anything right now.\n\nIn fact, I'm not even sure it buys you anything in the future. The thing \nis, SYN-flooding depends on overwhelming you with lots of simple packets. \nAnd since in the git protocol, the expense is not in the _packets_ but in \nthe server-side packing and data transfer, I don't see the point.\n\nIf you want to DoS a git pack server, you open a hundred _real_ git \nconnections to it, carefully selected so that they get unique packs (so \nthat the server can't cache them). You don't need to have some distributed \ndenial-of-service attack with lots of magic packets.\n\nThis is why the git daemon already limits the clients to 25 by default or \nsomething like that - it doesn't want to put too much strain on the \nserver.\n\nA much more important thing the git daemon could do is to kill connections \nfrom the same IP address when there's more than 25 pending ones. The \ndaemon actualy has the infrastructure for that - it's why it doesn't just \ncount its children, it actually saves child information away (it just \ndoesn't _use_ it for anything right now).\n\nSimilarly, git-upload-pack can be future-proofed by having it have some \ndata transfer timeout: if it doesn't make any progress at all in <n> \nseconds, just kill itself. Things like _that_ are likely to be a lot more \nimportant, I suspect.\n\nAnd no, I don't think th egit protocol should do authentication. It's \nhard. If you want to do authentication, you need to do encryption too, and \nthen you should do something else (but the git protocol _does_ work fine \nover an encyrpted channel, so the \"something else\" might be to have some \nsecure web interface tunnel protocol or similar, and then just support \n\"git over https\" or something ;).\n\n\t\t\tLinus\n"},{"id":"10361","messageId":"4357CB57.3070802@zytor.com","threadId":"2159","inReplyTo":"Pine.LNX.4.64.0510200924110.3369@g5.osdl.org","subject":"Re: Revamping the git protocol","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-20T16:52:39Z","receivedAt":"2005-10-20T16:52:39Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> Similarly, git-upload-pack can be future-proofed by having it have some \n> data transfer timeout: if it doesn't make any progress at all in <n> \n> seconds, just kill itself. Things like _that_ are likely to be a lot more \n> important, I suspect.\n> \n\nRight, I already submitted a patch for that.\n\n> And no, I don't think th egit protocol should do authentication. It's \n> hard. If you want to do authentication, you need to do encryption too, and \n> then you should do something else (but the git protocol _does_ work fine \n> over an encyrpted channel, so the \"something else\" might be to have some \n> secure web interface tunnel protocol or similar, and then just support \n> \"git over https\" or something ;).\n\ngit over ssh seems to be the obvious choice.\n\n\t-hpa\n"},{"id":"10362","messageId":"Pine.LNX.4.64.0510201001460.3369@g5.osdl.org","threadId":"2159","inReplyTo":"4357CB57.3070802@zytor.com","subject":"Re: Revamping the git protocol","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-20T17:17:34Z","receivedAt":"2005-10-20T17:17:34Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 20 Oct 2005, H. Peter Anvin wrote:\n> \n> git over ssh seems to be the obvious choice.\n\nYes, but Petr is right that there might be room for some lighter-weight \n\"gits\" secure protocol. One that doesn't necessarily require a whole user \nID thing.\n\nFor example, let's say that you're not the maintainer of your machine, but \nyou're in an environment where you are allowed to run daemons as yourself \n(at a university, for example). And you have a group of people who want to \nwork together at a project, but they don't want to give write permissions \nto the world or their bigger group (group \"student\").\n\nAnd git itself _does_ actually support that, already. You can use the \nstandard \"ssh:\" thing (or just \"hostname:pathname\"), and the GIT_SSH \nenvironment variable to set up any tunnelling program you want. Then you \ncan authenticate any way you want (and encrypt or not, whatever)..\n\nSo if somebody is in this situation, maybe we could have an example tunnel \nclient/server thing that does this.\n\nThis is unrelated to the git protocol itself, although the \"pack over \nssh/tunnel\" obviously uses all the same stuff for the actual transfer.\n\n(It might also be worthwhile to have .git/config specify what program to \nuse, so that you don't need a global environment variable. It might even \nbe per-host, ie we could have git-send-pack and git-fetch-pack understand \nconfig language like\n\n\t[connect]\n\t\tprogram=[server.uni.edu]:mytunnel\n\nor something. It shouldn't even be hard to do. Certainly simpler than \ndoing a good authenticating tunnel).\n\n\t\tLinus\n"},{"id":"10377","messageId":"Pine.LNX.4.63.0510210133060.10503@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"2159","inReplyTo":"Pine.LNX.4.64.0510201001460.3369@g5.osdl.org","subject":"Re: Revamping the git protocol","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-10-20T23:35:28Z","receivedAt":"2005-10-20T23:35:28Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 20 Oct 2005, Linus Torvalds wrote:\n\n> On Thu, 20 Oct 2005, H. Peter Anvin wrote:\n> > \n> > git over ssh seems to be the obvious choice.\n> \n> Yes, but Petr is right that there might be room for some lighter-weight \n> \"gits\" secure protocol. One that doesn't necessarily require a whole user \n> ID thing.\n> \n> For example, let's say that you're not the maintainer of your machine, but \n> you're in an environment where you are allowed to run daemons as yourself \n> (at a university, for example). And you have a group of people who want to \n> work together at a project, but they don't want to give write permissions \n> to the world or their bigger group (group \"student\").\n\nIf you are not maintainer, you could still start an SSH daemon which \nlistens on a port>1024 and gets its password data from a file different \nfrom /etc/shadow (You could even use PAM...).\n\nCiao,\nDscho\n"},{"id":"10394","messageId":"20051021010429.GC30889@pasky.or.cz","threadId":"2159","inReplyTo":"4357BCBC.1020706@zytor.com","subject":"Re: Revamping the git protocol","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-10-21T01:04:29Z","receivedAt":"2005-10-21T01:04:29Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Thu, Oct 20, 2005 at 05:50:20PM CEST, I got a letter\nwhere \"H. Peter Anvin\" <hpa@zytor.com> told me that...\n> Another issue is that currently there is no error information propagated \n> back to the client; the server logs an error in its own logs, but the \n> client is simply disconnected.\n\nYes. I agree that while it seems quite complex compared to what we have\nnow, your proposal has good points. But if we are going with the\nchallenge-response at all and if we are going with the simple form,\nI was merely trying to make sure that it is as compatible as possible.\n\n> Anyone noticed that either of those schemes aren't actually \n> backward-compatible in any way (old client talking to new server will be \n> disconnected), and that unfortunately is the best thing one can do with \n> the current setup, exactly because there is no option negotiation phase?\n\nYes, option negotiation would solve this for us. But my scheme _is_\nbackwards-compatible in the way that new client taking to old server\nwill not be disconnected, so it's 50% better than the original proposal.\n\nBut I think that considering the long run, we should either not do this\nchallenge-response thing at all, and fix the problem by other (Linus')\nmeans, or go for the \"complex\" scheme. I'd prefer the latter - sending\nthe error messages to the client alone is a huge improvement.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"}]}