{"thread":{"id":"19323","subject":"Request for detailed documentation of git pack protocol","startedAt":"2009-05-12T21:29:14Z","lastAt":"2009-06-13T09:30:55Z","messageCount":66,"participants":["Jakub Narebski","Shawn O. Pearce","Scott Chacon","Andreas Ericsson","Nicolas Pitre","A Large Angry SCM","Clemens Buchacher","Ealdwulf Wuffinga","Linus Torvalds","Junio C Hamano","Robin H. Johnson","Johannes Sixt","Tony Finch"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"113694","messageId":"200905122329.15379.jnareb@gmail.com","threadId":"19323","inReplyTo":null,"subject":"Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-12T21:29:14Z","receivedAt":"2009-05-12T21:29:14Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"We have now proliferation of different (re)implementations of git:\nJGit in Java, Dulwich in Python, Grit in Ruby; and there are other\nplanned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\nin Objective-C (for iPhone IIRC). At some time they would reach\nthe point (or reached it already) of implementing git-daemon...\nbut currently the documentation of git protocol is lacking.\n\nThis can lead, as you can read from recent post on git mailing, to\nimplementing details wrong (like Dulwich not using full SHA-1 where\nit should, leading to ordinary git clients to failing to fetch from it),\nor fail at best practices of implementation (like JGit last issue with\ndeadlocking for multi_ack extension).\n\nThe current documentation of git protocol is very sparse; the docs\nin Documentation/technical/pack-protocol.txt offer only a sketch of\nexchange. You can find more, including pkt-line format, a way sideband\nis multiplexed, and how capabilities are negotiated between server and\nclient in design document for \"smart\" HTTP server, for example in\n Subject: Re: More on git over HTTP POST\n Message-ID: <20080803025602.GB27465@spearce.org>\n URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n\nIt would be really nice, I think, to have RFC for git pack protocol.\nAnd it would help avoid incompatibilities between different clients\nand servers. If the document would contain expected behaviour of\nclient and server and Best Current Practices it would help avoid\npitfals when implementing git-daemon in other implementation.\n\nPerhaps in the future it could be sent for inclusion as \"official\"\nRFC to IETF? (Dreaming big).\n\n\nUnfortunately I don't know enough about this area of code to write\none myself.\n\n-- \nJakub Narebski\nShadeHawk or jnareb on #git\nPoland\n"},{"id":"113711","messageId":"20090512233450.GY30527@spearce.org","threadId":"19323","inReplyTo":"200905122329.15379.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-12T23:34:50Z","receivedAt":"2009-05-12T23:34:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> We have now proliferation of different (re)implementations of git:\n> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n> in Objective-C (for iPhone IIRC). At some time they would reach\n> the point (or reached it already) of implementing git-daemon...\n> but currently the documentation of git protocol is lacking.\n\nWell, lets see...\n\nJGit - me and Robin, here on git ML. JGit is the closest\nreimplementation effort to the canonical C implementation.\nJGit runs in production servers for many folks, e.g. quite\na few Google engineers use the JGit server every day. Its\nour main git daemon.\n\nGrit - GitHub folks. They know where to find us. And their\nbusiness is Git. If Grit isn't right, they'll make it right,\nor possibly suffer a loss of customers. I'm fairly certain\nthat GitHub runs Grit in production.\n\nObjectGit - Scott Chacon, again, a GitHub folk. Though he has\nexpressed interest in moving to JGit or libgit2 where/when possible.\n\nDulwich - off in its own world and not even trying to match basic\nprotocol rules by just watching what happens when you telnet to a\ngit port. No clue how that's going to fair.\n\ngit# - We'll see. Mono GSoC Git projects have a really bad\nreputation of ignoring the existing git knowledge and hoping\nthey can invent the wheel on their own.\n\n> This can lead, as you can read from recent post on git mailing, to\n> implementing details wrong (like Dulwich not using full SHA-1 where\n> it should, leading to ordinary git clients to failing to fetch from it),\n> or fail at best practices of implementation (like JGit last issue with\n> deadlocking for multi_ack extension).\n\nDulwich is just busted.\n\nNo existing developers knew that the fetch-pack/upload-pack protocol\nhas this required implicit buffering consideration until JGit\ndeadlocked over it. But even then I'm still not 100% sure this\nis true, or if it is just an artifact of the JGit upload-pack side\nimplementation being partially wrong.\n\n> The current documentation of git protocol is very sparse; the docs\n> in Documentation/technical/pack-protocol.txt offer only a sketch of\n> exchange. You can find more, including pkt-line format, a way sideband\n> is multiplexed, and how capabilities are negotiated between server and\n> client in design document for \"smart\" HTTP server, for example in\n> Subject: Re: More on git over HTTP POST\n> Message-ID: <20080803025602.GB27465@spearce.org>\n> URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n\nSeriously? Don't link to that. Its a horrible version of the smart\nHTTP RFC, and worse, it doesn't describe what you say it describes.\n \n> It would be really nice, I think, to have RFC for git pack protocol.\n> And it would help avoid incompatibilities between different clients\n> and servers. If the document would contain expected behaviour of\n> client and server and Best Current Practices it would help avoid\n> pitfals when implementing git-daemon in other implementation.\n\nYea, it would be nice. But find me someone who knows the protocol\nand who has the time to document the #!@* thing. Maybe I'll try\nto work on this myself, but I'm strapped for time, especially over\nthe next two-to-three months.\n\nAnd lets not even start to mention Dulwich not completing a thin\npack before storing it on disk. Those sorts of on disk things\nmatter to other more popular Git implementations (c git, jgit).\n \n-- \nShawn.\n"},{"id":"113907","messageId":"200905141024.17525.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090512233450.GY30527@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-14T08:24:15Z","receivedAt":"2009-05-14T08:24:15Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 13 May 2009 Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n\n> > We have now proliferation of different (re)implementations of git:\n> > JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n> > planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n> > in Objective-C (for iPhone IIRC). At some time they would reach\n> > the point (or reached it already) of implementing git-daemon...\n> > but currently the documentation of git protocol is lacking.\n> \n> Well, lets see...\n> \n> JGit - me and Robin, here on git ML. JGit is the closest\n> reimplementation effort to the canonical C implementation.\n> JGit runs in production servers for many folks, e.g. quite\n> a few Google engineers use the JGit server every day. Its\n> our main git daemon.\n> \n> Grit - GitHub folks. They know where to find us. And their\n> business is Git. If Grit isn't right, they'll make it right,\n> or possibly suffer a loss of customers. I'm fairly certain\n> that GitHub runs Grit in production.\n> \n> ObjectGit - Scott Chacon, again, a GitHub folk. Though he has\n> expressed interest in moving to JGit or libgit2 where/when possible.\n> \n> Dulwich - off in its own world and not even trying to match basic\n> protocol rules by just watching what happens when you telnet to a\n> git port. No clue how that's going to fair.\n> \n> git# - We'll see. Mono GSoC Git projects have a really bad\n> reputation of ignoring the existing git knowledge and hoping\n> they can invent the wheel on their own.\n\nSo you are saying that even if detailed pack protocol specification\nisn't written down (Documentation/technical/pack-protocol.txt is more\nof a sketch than reference documentation), the knowledge is there,\nand it is not that hard to get (just ask on git mailing list), isn't it?\n \n> > This can lead, as you can read from recent post on git mailing, to\n> > implementing details wrong (like Dulwich not using full SHA-1 where\n> > it should, leading to ordinary git clients to failing to fetch from it),\n> > or fail at best practices of implementation (like JGit last issue with\n> > deadlocking for multi_ack extension).\n> \n> Dulwich is just busted.\n\nThat was my impression too. Those details it got wrong aren't so\nobscure and hard to get right...\n\n> \n> No existing developers knew that the fetch-pack/upload-pack protocol\n> has this required implicit buffering consideration until JGit\n> deadlocked over it. But even then I'm still not 100% sure this\n> is true, or if it is just an artifact of the JGit upload-pack side\n> implementation being partially wrong.\n\nWell... I guess that section on Best Current Practices to avoid \ndeadlocking would not be there to avoid this issue in JGit, but\nwould be added for the future later.\n\n> \n> > The current documentation of git protocol is very sparse; the docs\n> > in Documentation/technical/pack-protocol.txt offer only a sketch of\n> > exchange. You can find more, including pkt-line format, a way sideband\n> > is multiplexed, and how capabilities are negotiated between server and\n> > client in design document for \"smart\" HTTP server, for example in\n> > Subject: Re: More on git over HTTP POST\n> > Message-ID: <20080803025602.GB27465@spearce.org>\n> > URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n> \n> Seriously? Don't link to that. Its a horrible version of the smart\n> HTTP RFC, and worse, it doesn't describe what you say it describes.\n\nOoops, I am sorry. This was my bookmark into this thread (which is very\ninteresting, and contain host of otherwise unknown to me information\nabout pack protocol), but the post in this thread was quite arbitrary\n(a random post where I decided that this thread is interesting enough\nto bookmark, and long enough to not want to save all interesting posts).\n\n> \n> > It would be really nice, I think, to have RFC for git pack protocol.\n> > And it would help avoid incompatibilities between different clients\n> > and servers. If the document would contain expected behaviour of\n> > client and server and Best Current Practices it would help avoid\n> > pitfals when implementing git-daemon in other implementation.\n> \n> Yea, it would be nice. But find me someone who knows the protocol\n> and who has the time to document the #!@* thing. Maybe I'll try\n> to work on this myself, but I'm strapped for time, especially over\n> the next two-to-three months.\n\nI was afraid of this: that the people who know pack protocol good\nenough to be able to write it down are otherwise busy. But we get\ndetailed / updated packfile and pack index format descriptions some\ntime ago (thanks all that contributed to it!). I hope that the same\nwould happen with pack _protocol_ description.\n\nI was hoping of document in RFC format; dreaming about having it\nsubmitted to IETF as (at least) unofficial RFC like Atom Publication\nProtocol (or is it proper RFC these days?), and then accepted like\nHTTP protocol. But I understand that it is not the same situation;\nthere wouldn't (and perhaps shouldn't) too many independent git-daemon\nimplementations...\n\n> \n> And lets not even start to mention Dulwich not completing a thin\n> pack before storing it on disk. Those sorts of on disk things\n> matter to other more popular Git implementations (c git, jgit).\n\nUgh! Errr... aren't thin packs send only if other side has the\ncapability for it? What is then Dulwich doing announcing such \ncapability when not supporting it correctly...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"113931","messageId":"d411cc4a0905140655y244f21aem44f1e246dd74d80c@mail.gmail.com","threadId":"19323","inReplyTo":"20090512233450.GY30527@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-05-14T13:55:39Z","receivedAt":"2009-05-14T13:55:39Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nOn Tue, May 12, 2009 at 4:34 PM, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n>> We have now proliferation of different (re)implementations of git:\n>> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n>> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n>> in Objective-C (for iPhone IIRC).  At some time they would reach\n>> the point (or reached it already) of implementing git-daemon...\n>> but currently the documentation of git protocol is lacking.\n>\n> Well, lets see...\n>\n> JGit - me and Robin, here on git ML.  JGit is the closest\n> reimplementation effort to the canonical C implementation.\n> JGit runs in production servers for many folks, e.g. quite\n> a few Google engineers use the JGit server every day.  Its\n> our main git daemon.\n>\n> Grit - GitHub folks.  They know where to find us.  And their\n> business is Git.  If Grit isn't right, they'll make it right,\n> or possibly suffer a loss of customers.  I'm fairly certain\n> that GitHub runs Grit in production.\n>\n> ObjectGit - Scott Chacon, again, a GitHub folk.  Though he has\n> expressed interest in moving to JGit or libgit2 where/when possible.\n\nActually, all of this work has moved to CocoaGit, which is much\nfarther along than ObjectiveGit ever was. Although I would love to\nuse libgit2 when it gets that far, this was for Mac/iPhone native\nclient work which JGit would not be helpful for.\n\n>\n> Dulwich - off in its own world and not even trying to match basic\n> protocol rules by just watching what happens when you telnet to a\n> git port.  No clue how that's going to fair.\n\nOddly enough, I'm in Dulwich land too. I've been working on a\nMercurial plugin that will provide a two way lossless bridge for Hg to\nbe able to push and pull to/from a Git server. I've fixed some of the\nissues I've found with the client side work and both pushes and pulls\nwill work now. (I did turn off 'thin-pack' capability announcement,\nsince you're correct that it simply was not properly implemented).\n\nIf we're going to round out the list, I've also worked on an\nActionScript partial implementation, but it never got to the packfile\nlevel, and some of the Erlang guys are interested in writing at least\na partial Erlang implementation too, which I may get involved in at\nsome point.\n\nIt seems like if anyone would do what you're asking, it's probably me.\nIn the next few weeks, I do what I can to fix up the remainder of the\nDulwich code as part of my hg-git work. I'm also working with Shawn\non the Apress book, where I was going to try to document much of this\ninformation, perhaps I could try writing an RFC as an appendix or\nsomething - then that will force him to spend time correcting\neverything I got wrong :) At least that might be a good starting\npoint - I'm unfamiliar with the actual RFC process, so I'll research\nthat a bit today. I don't mind writing it, I think it would be really\nreally useful to have, I just am unfamiliar with the process.\n\nThanks,\nScott\n\n\n>\n> git# - We'll see.  Mono GSoC Git projects have a really bad\n> reputation of ignoring the existing git knowledge and hoping\n> they can invent the wheel on their own.\n>\n>> This can lead, as you can read from recent post on git mailing, to\n>> implementing details wrong (like Dulwich not using full SHA-1 where\n>> it should, leading to ordinary git clients to failing to fetch from it),\n>> or fail at best practices of implementation (like JGit last issue with\n>> deadlocking for multi_ack extension).\n>\n> Dulwich is just busted.\n>\n> No existing developers knew that the fetch-pack/upload-pack protocol\n> has this required implicit buffering consideration until JGit\n> deadlocked over it.  But even then I'm still not 100% sure this\n> is true, or if it is just an artifact of the JGit upload-pack side\n> implementation being partially wrong.\n>\n>> The current documentation of git protocol is very sparse; the docs\n>> in Documentation/technical/pack-protocol.txt offer only a sketch of\n>> exchange.  You can find more, including pkt-line format, a way sideband\n>> is multiplexed, and how capabilities are negotiated between server and\n>> client in design document for \"smart\" HTTP server, for example in\n>>   Subject: Re: More on git over HTTP POST\n>>   Message-ID: <20080803025602.GB27465@spearce.org>\n>>   URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n>\n> Seriously?  Don't link to that.  Its a horrible version of the smart\n> HTTP RFC, and worse, it doesn't describe what you say it describes.\n>\n>> It would be really nice, I think, to have RFC for git pack protocol.\n>> And it would help avoid incompatibilities between different clients\n>> and servers.  If the document would contain expected behaviour of\n>> client and server and Best Current Practices it would help avoid\n>> pitfals when implementing git-daemon in other implementation.\n>\n> Yea, it would be nice.  But find me someone who knows the protocol\n> and who has the time to document the #!@* thing.  Maybe I'll try\n> to work on this myself, but I'm strapped for time, especially over\n> the next two-to-three months.\n>\n> And lets not even start to mention Dulwich not completing a thin\n> pack before storing it on disk.  Those sorts of on disk things\n> matter to other more popular Git implementations (c git, jgit).\n>\n> --\n> Shawn.\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"113934","messageId":"20090514144433.GD30527@spearce.org","threadId":"19323","inReplyTo":"d411cc4a0905140655y244f21aem44f1e246dd74d80c@mail.gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-14T14:44:34Z","receivedAt":"2009-05-14T14:44:34Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Scott Chacon wrote:\n> On Tue, May 12, 2009 at 4:34 PM, Shawn O. Pearce wrote:\n> >\n> > Dulwich - off in its own world and not even trying to match basic\n> > protocol rules by just watching what happens when you telnet to a\n> > git port. ??No clue how that's going to fair.\n> \n> Oddly enough, I'm in Dulwich land too. I've been working on a\n> Mercurial plugin that will provide a two way lossless bridge for Hg to\n> be able to push and pull to/from a Git server.\n\nHow are you going to represent an n-way merge in Git in Hg?\n\n> I've fixed some of the\n> issues I've found with the client side work and both pushes and pulls\n> will work now. (I did turn off 'thin-pack' capability announcement,\n> since you're correct that it simply was not properly implemented).\n\nI'm half interested in Dulwich for \"repo\"[1] but I need the #@!*\nlibrary to be stable and correctly implement Git conventions. :-)\n\n[1] http://android.git.kernel.org/?p=tools/repo.git;a=summary\n\n> If we're going to round out the list, I've also worked on an\n> ActionScript partial implementation, but it never got to the packfile\n> level,\n\nActionScript? WTF? As in that thing that embeds in Flash?\n\n> and some of the Erlang guys are interested in writing at least\n> a partial Erlang implementation too, which I may get involved in at\n> some point.\n\nI heard they moved their official repository to Git. Their VM as\na network server is just plain awesome. I half wish I was using\nthat for the Gerrit backend rather than Apache MINA. Erlang is\nrock-solid and doesn't have major threading bugs in its core.\n\nOh heck, I just found the documentation for the Erlang sshd. Nice.\nSadly it lacks public key support it seems, and a solid Git library\nwith server protocols fully implemented, but, eh, its management\nis way better than Java.\n\n> It seems like if anyone would do what you're asking, it's probably me.\n> In the next few weeks, I do what I can to fix up the remainder of the\n> Dulwich code as part of my hg-git work. I'm also working with Shawn\n> on the Apress book, where I was going to try to document much of this\n> information, perhaps I could try writing an RFC as an appendix or\n> something - then that will force him to spend time correcting\n> everything I got wrong :)\n\nHah!\n\nEven if you don't write it for the book, I'll certainly try to\ngive a technical review over the content. That goes for anyone\nwho takes the time to write the protocol out, and has a fair clue\nas to how it currently works.\n\n-- \nShawn.\n"},{"id":"113939","messageId":"20090514145724.GE30527@spearce.org","threadId":"19323","inReplyTo":"200905141024.17525.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-14T14:57:24Z","receivedAt":"2009-05-14T14:57:24Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Wed, 13 May 2009 Shawn O. Pearce wrote:\n> > Jakub Narebski wrote:\n> > > We have now proliferation of different (re)implementations of git:\n> > > JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n> > > planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n> > > in Objective-C (for iPhone IIRC). At some time they would reach\n> > > the point (or reached it already) of implementing git-daemon...\n> > > but currently the documentation of git protocol is lacking.\n> > \n> > Well, lets see...\n> \n> So you are saying that even if detailed pack protocol specification\n> isn't written down (Documentation/technical/pack-protocol.txt is more\n> of a sketch than reference documentation), the knowledge is there,\n> and it is not that hard to get (just ask on git mailing list), isn't it?\n\nYup. We've tried to keep JGit right here on this list just to keep\nthe knowledge concentrated here, so git@vger is the place anyone\ncan ask questions, and get good answers.\n\n> > No existing developers knew that the fetch-pack/upload-pack protocol\n> > has this required implicit buffering consideration until JGit\n> > deadlocked over it. But even then I'm still not 100% sure this\n> > is true, or if it is just an artifact of the JGit upload-pack side\n> > implementation being partially wrong.\n> \n> Well... I guess that section on Best Current Practices to avoid \n> deadlocking would not be there to avoid this issue in JGit, but\n> would be added for the future later.\n\nSo I'm actually right (and Junio confirmed it off list), the\nfetch-pack/upload-pack protocol with multi_ack enabled requires\na buffer on the client side of at least 2952 bytes which can be\ndrained to the server after the client enters its receive phase.\n\nIn practical implementations like git:// TCP and SSH, there is\nenough inherit buffering in the TX side of the client that this\nisn't an issue.\n\nIn loopback mode for local file URIs, it may become an issue. C Git\nis just getting lucky by the pipe size I think. Though I thought I\nread somewhere yesterday pipe FIFOs in Linux were being allocated\nat 512 bytes, not one system page. Of course other systems could\nallocate whatever size they want too, and may allocate something\nbelow the 2952 minimum, and we'd most likely see a deadlock on them.\n\n> > > The current documentation of git protocol is very sparse; the docs\n> > > in Documentation/technical/pack-protocol.txt offer only a sketch of\n> > > exchange. You can find more, including pkt-line format, a way sideband\n> > > is multiplexed, and how capabilities are negotiated between server and\n> > > client in design document for \"smart\" HTTP server, for example in\n> > > Subject: Re: More on git over HTTP POST\n> > > Message-ID: <20080803025602.GB27465@spearce.org>\n> > > URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n> > \n> > Seriously? Don't link to that. Its a horrible version of the smart\n> > HTTP RFC, and worse, it doesn't describe what you say it describes.\n> \n> Ooops, I am sorry. This was my bookmark into this thread (which is very\n> interesting, and contain host of otherwise unknown to me information\n> about pack protocol), but the post in this thread was quite arbitrary\n> (a random post where I decided that this thread is interesting enough\n> to bookmark, and long enough to not want to save all interesting posts).\n\nGMane is dead right now, otherwise I'd try to find the link you\nwere more likely talking about. I think you were right, there may\nhave been a much better post in that particular thread.\n \n> > And lets not even start to mention Dulwich not completing a thin\n> > pack before storing it on disk. Those sorts of on disk things\n> > matter to other more popular Git implementations (c git, jgit).\n> \n> Ugh! Errr... aren't thin packs send only if other side has the\n> capability for it?\n\nYes.\n\n> What is then Dulwich doing announcing such \n> capability when not supporting it correctly...\n\nBecause the implementation is just busted.\n\n-- \nShawn.\n"},{"id":"113942","messageId":"200905141701.41212.jnareb@gmail.com","threadId":"19323","inReplyTo":"d411cc4a0905140655y244f21aem44f1e246dd74d80c@mail.gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-14T15:01:40Z","receivedAt":"2009-05-14T15:01:40Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 14 May 2009, Scott Chacon wrote:\n> On Tue, May 12, 2009 at 4:34 PM, Shawn O. Pearce wrote:\n>> Jakub Narebski wrote:\n\n>>> We have now proliferation of different (re)implementations of git:\n>>> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n>>> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n>>> in Objective-C (for iPhone IIRC).  At some time they would reach\n>>> the point (or reached it already) of implementing git-daemon...\n>>> but currently the documentation of git protocol is lacking.\n>>\n>> Well, lets see...\n\n[...]\n>> ObjectGit - Scott Chacon, again, a GitHub folk.  Though he has\n>> expressed interest in moving to JGit or libgit2 where/when possible.\n> \n> Actually, all of this work has moved to CocoaGit, which is much\n> farther along than ObjectiveGit ever was. Although I would love to\n> use libgit2 when it gets that far, this was for Mac/iPhone native\n> client work which JGit would not be helpful for.\n\nCould you give URL for homepage or announcement (if it exists), and\nfor git repository / web interface page for CocoaGit? It isn't present\non http://git.or.cz/gitwiki/InterfacesFrontendsAndTools which tries\nto be clearinghouse and list all significant git tools... but which\nis probably hopelessly out of date now (unfortunately).\n\n> \n>>\n>> Dulwich - off in its own world and not even trying to match basic\n>> protocol rules by just watching what happens when you telnet to a\n>> git port.  No clue how that's going to fair.\n> \n> Oddly enough, I'm in Dulwich land too. I've been working on a\n> Mercurial plugin that will provide a two way lossless bridge for Hg to\n> be able to push and pull to/from a Git server.\n\nI'm assuming here that the bridge has to remember somehow about the\ninfo which cannot be represented in other SCM (like octopus merges,\nor tag objects, or tagging non-commits in Git; like I guess 'rename\ntracking' information in Mercurial) to be it truly two-way...\n\n> I've fixed some of the \n> issues I've found with the client side work and both pushes and pulls\n> will work now. (I did turn off 'thin-pack' capability announcement,\n> since you're correct that it simply was not properly implemented).\n> \n> If we're going to round out the list, I've also worked on an\n> ActionScript partial implementation, but it never got to the packfile\n> level, and some of the Erlang guys are interested in writing at least\n> a partial Erlang implementation too, which I may get involved in at\n> some point.\n\nWell, with yet another implementation it is even more important to have\ngood technical documentation of file formats and network protocols.\n\nBTW. if I remember correctly there were some hobbyist one-person \n(single-developer) implementations of git in Haskell and in Lisp\nor Scheme...\n\n> \n> It seems like if anyone would do what you're asking, it's probably me.\n> In the next few weeks, I do what I can to fix up the remainder of the\n> Dulwich code as part of my hg-git work. I'm also working with Shawn\n> on the Apress book, where I was going to try to document much of this\n> information, perhaps I could try writing an RFC as an appendix or\n> something - then that will force him to spend time correcting\n> everything I got wrong :) At least that might be a good starting\n> point - I'm unfamiliar with the actual RFC process, so I'll research\n> that a bit today. I don't mind writing it, I think it would be really\n> really useful to have, I just am unfamiliar with the process.\n\nI don't think RFC _process_ is something to worry about; in the future\nperhaps (just like Atom Publishing protocol was submitted to IETF).\nI was thinking about _format_ used in RFC (BNF-like specification,\nspecific semantic for 'MUST' etc. like in RFC2119). Although any format\n(more or less formal) would be better that none.\n\nThank you very much for your offer!\n\n-- \nJakub Narebski\nPoland\n"},{"id":"113940","messageId":"4A0C3282.4000101@op5.se","threadId":"19323","inReplyTo":"20090514145724.GE30527@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-14T15:02:26Z","receivedAt":"2009-05-14T15:02:26Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> \n> In loopback mode for local file URIs, it may become an issue. C Git\n> is just getting lucky by the pipe size I think. Though I thought I\n> read somewhere yesterday pipe FIFOs in Linux were being allocated\n> at 512 bytes, not one system page. Of course other systems could\n> allocate whatever size they want too, and may allocate something\n> below the 2952 minimum, and we'd most likely see a deadlock on them.\n> \n\nLinux allocates one page 4096 bytes for a FIFO. 512 is the maximum\nsize guaranteed by POSIX to result in an atomic write.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"113945","messageId":"alpine.LFD.2.00.0905141353040.6741@xanadu.home","threadId":"19323","inReplyTo":"200905141024.17525.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-14T18:13:21Z","receivedAt":"2009-05-14T18:13:21Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 14 May 2009, Jakub Narebski wrote:\n\n> I was afraid of this: that the people who know pack protocol good\n> enough to be able to write it down are otherwise busy. But we get\n> detailed / updated packfile and pack index format descriptions some\n> time ago (thanks all that contributed to it!). I hope that the same\n> would happen with pack _protocol_ description.\n\nIf someone with the wish for such a document volunteers to work on it \nthen I'm sure people with fuller knowledge will review and comment on \nthe result as appropriate.\n\n> I was hoping of document in RFC format; dreaming about having it\n> submitted to IETF as (at least) unofficial RFC like Atom Publication\n> Protocol (or is it proper RFC these days?), and then accepted like\n> HTTP protocol.\n\nI think we'd have to move to a new version of the protocol for that. \nThe current protocol, even if it does the job, is not particularly \nelegant.\n\n> > And lets not even start to mention Dulwich not completing a thin\n> > pack before storing it on disk. Those sorts of on disk things\n> > matter to other more popular Git implementations (c git, jgit).\n> \n> Ugh! Errr... aren't thin packs send only if other side has the\n> capability for it? What is then Dulwich doing announcing such \n> capability when not supporting it correctly...\n\nThey probably don't bother because in theory you don't need to complete \na thin pack for the system to still work. We require that any pack \nnever contain a delta which base object is in a different pack because \nthat makes for better performances when accessing the pack and when \nrepacking. And not doing so makes pack validation (think verify-pack) \nimpossible without the dependent objects, and that makes incremental \nrepacking much much harder wrt prevention of delta cycles.\n\nThose validation tools from C git (fsck, verify-pack, etc.) should be \nquite useful for people wishing to implement their own git.\n\n\nNicolas\n"},{"id":"113959","messageId":"200905142227.10669.jnareb@gmail.com","threadId":"19323","inReplyTo":"alpine.LFD.2.00.0905141353040.6741@xanadu.home","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-14T20:27:07Z","receivedAt":"2009-05-14T20:27:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 14 May 2009, Nicolas Pitre wrote:\n> On Thu, 14 May 2009, Jakub Narebski wrote:\n> \n> > I was afraid of this: that the people who know pack protocol good\n> > enough to be able to write it down are otherwise busy. But we get\n> > detailed / updated packfile and pack index format descriptions some\n> > time ago (thanks all that contributed to it!). I hope that the same\n> > would happen with pack _protocol_ description.\n> \n> If someone with the wish for such a document volunteers to work on it \n> then I'm sure people with fuller knowledge will review and comment on \n> the result as appropriate.\n\nWell, but still somebody with time and at least some expertise in\nthe area would be required to start it.\n \n> > I was hoping of document in RFC format; dreaming about having it\n> > submitted to IETF as (at least) unofficial RFC like Atom Publication\n> > Protocol (or is it proper RFC these days?), and then accepted like\n> > HTTP protocol.\n> \n> I think we'd have to move to a new version of the protocol for that. \n> The current protocol, even if it does the job, is not particularly \n> elegant.\n\nAre all RFC (including proposals / informational RFCs) defined protocols\nelegant? Well... perhaps they are. The quality of IETF standards is way\nhigher than, say, ECMA :-)\n\nBut I accept that having RFC to be on the list of 'official' RFCs, even\nas an \"experimental\" RFC is just a dream. Nevertheless I think that \nfollowing RFC format, which includes using a common set of terms such \nas \"MUST\" and \"NOT RECOMMENDED\" (as defined by RFC 2119), Augmented \nBackus–Naur Form (ABNF) (as defined by RFC 5234) as a metalanguage,\nwould be a good idea for technical / protocol documentation.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"113981","messageId":"4A0CBE35.6030004@gmail.com","threadId":"19323","inReplyTo":"200905141701.41212.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2009-05-15T00:58:29Z","receivedAt":"2009-05-15T00:58:29Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Jakub Narebski wrote:\n> I don't think RFC _process_ is something to worry about; in the future\n> perhaps (just like Atom Publishing protocol was submitted to IETF).\n> I was thinking about _format_ used in RFC (BNF-like specification,\n> specific semantic for 'MUST' etc. like in RFC2119). Although any format\n> (more or less formal) would be better that none.\n\nStandardese, the peculiar dialect and formalism employed by RFC authors, \nis not difficult to master. The difficult part is writing the prose \nthat's an _unambiguous_ description of the protocol you're attempting to \ndocument. There's even a tool, xml2rfc, that will do the formatting for you.\n"},{"id":"114002","messageId":"20090515165155.GA13177@localhost","threadId":"19323","inReplyTo":"20090514145724.GE30527@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2009-05-15T16:51:55Z","receivedAt":"2009-05-15T16:51:55Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Thu, May 14, 2009 at 07:57:24AM -0700, Shawn O. Pearce wrote:\n> > > Jakub Narebski wrote:\n> > > > The current documentation of git protocol is very sparse; the docs\n> > > > in Documentation/technical/pack-protocol.txt offer only a sketch of\n> > > > exchange. You can find more, including pkt-line format, a way sideband\n> > > > is multiplexed, and how capabilities are negotiated between server and\n> > > > client in design document for \"smart\" HTTP server, for example in\n> > > > Subject: Re: More on git over HTTP POST\n> > > > Message-ID: <20080803025602.GB27465@spearce.org>\n> > > > URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196\n[...]\n> GMane is dead right now, otherwise I'd try to find the link you\n> were more likely talking about. I think you were right, there may\n> have been a much better post in that particular thread.\n\nI believe this is the most recent version:\n\n\thttp://article.gmane.org/gmane.comp.version-control.git/94313\n"},{"id":"114017","messageId":"efe2b6d70905151205r70a167a4kdc3575f4c0ec9574@mail.gmail.com","threadId":"19323","inReplyTo":"4A0CBE35.6030004@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Ealdwulf Wuffinga","fromEmail":"ealdwulf@googlemail.com","sentAt":"2009-05-15T19:05:10Z","receivedAt":"2009-05-15T19:05:10Z","isPatch":false,"sender":{"key":"ealdwulf@googlemail.com","avatar":null},"body":"On Fri, May 15, 2009 at 1:58 AM, A Large Angry SCM wrote:\n\n> Standardese, the peculiar dialect and formalism employed by RFC authors, is\n> not difficult to master. The difficult part is writing the prose that's an\n> _unambiguous_ description of the protocol you're attempting to document.\n> There's even a tool, xml2rfc, that will do the formatting for you.\n\nThere are also tools for writing unambiguous prose:\nhttp://en.wikipedia.org/wiki/Controlled_natural_language.\nThey look potentially tedious to use, but I haven't actually tried.\n\nEaldwulf\n"},{"id":"114030","messageId":"alpine.LFD.2.01.0905151324280.3343@localhost.localdomain","threadId":"19323","inReplyTo":"4A0C3282.4000101@op5.se","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-15T20:29:06Z","receivedAt":"2009-05-15T20:29:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 14 May 2009, Andreas Ericsson wrote:\n\n> Shawn O. Pearce wrote:\n> > \n> > In loopback mode for local file URIs, it may become an issue. C Git\n> > is just getting lucky by the pipe size I think. Though I thought I\n> > read somewhere yesterday pipe FIFOs in Linux were being allocated\n> > at 512 bytes, not one system page. Of course other systems could\n> > allocate whatever size they want too, and may allocate something\n> > below the 2952 minimum, and we'd most likely see a deadlock on them.\n> > \n> \n> Linux allocates one page 4096 bytes for a FIFO. 512 is the maximum\n> size guaranteed by POSIX to result in an atomic write.\n\nActually, modern Linux will allocate up to 16 pages (PIPE_BUFFERS), but \nthey may not all be filled - we coalesce small writes only if the end \nresult fits entirely into a page. So the maximum buffer is 16*PAGE_SIZE, \nand the minimum buffer space (assuming regular \"write()\" system calls) is \nsomething like 16*(PAGE_SIZE/2+1).\n\nBut yeah, POSIX allows for much smaller buffers.\n\n\t\t\tLinus\n"},{"id":"115284","messageId":"200906022339.08639.jnareb@gmail.com","threadId":"19323","inReplyTo":"d411cc4a0905140655y244f21aem44f1e246dd74d80c@mail.gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-02T21:39:07Z","receivedAt":"2009-06-02T21:39:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 14 May 2009, Scott Chacon wrote:\n> On Tue, May 12, 2009 at 4:34 PM, Shawn O. Pearce wrote:\n>> Jakub Narebski wrote:\n\n>>> We have now proliferation of different (re)implementations of git:\n>>> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other\n>>> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit\n>>> in Objective-C (for iPhone IIRC).  At some time they would reach\n>>> the point (or reached it already) of implementing git-daemon...\n>>> but currently the documentation of git protocol is lacking.\n[...]\n \n> It seems like if anyone would do what you're asking, it's probably me.\n> [...] I'm also working with Shawn\n> on the Apress book, where I was going to try to document much of this\n> information, perhaps I could try writing an RFC as an appendix or\n> something - then that will force him to spend time correcting\n> everything I got wrong :) At least that might be a good starting\n> point - I'm unfamiliar with the actual RFC process, so I'll research\n> that a bit today. I don't mind writing it, I think it would be really\n> really useful to have, I just am unfamiliar with the process.\n\n[...]\n>>> It would be really nice, I think, to have RFC for git pack protocol.\n>>> And it would help avoid incompatibilities between different clients\n>>> and servers.  If the document would contain expected behaviour of\n>>> client and server and Best Current Practices it would help avoid\n>>> pitfals when implementing git-daemon in other implementation.\n>>\n>> Yea, it would be nice.  But find me someone who knows the protocol\n>> and who has the time to document the #!@* thing.  Maybe I'll try\n>> to work on this myself, but I'm strapped for time, especially over\n>> the next two-to-three months.\n\nI see that there is (at least beginnings of) description of git pack\nprotocol in section \"Transfer Protocols\"[1][2] of chapter \"7. Internals\nand Plumbing\" of \"Git Community Book\".\n\n [1] http://book.git-scm.com/7_transfer_protocols.html\n [2] http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\n\nLet me quote here relevant part of this chapter, with some comment I am\nnot sure validity of... and therefore I'd like to ask for comments here,\nrather than sending a patch of pull request already\n\n\n> ### Fetching Data with Upload Pack ###\n>\n> For the smarter protocols, fetching objects is much more efficient. \n> A socket is opened, either over ssh or over port 9418 (in the case of\n> the git:// protocol), and the linkgit:git-fetch-pack[1] command on\n> the client begins communicating with a forked\n> linkgit:git-upload-pack[1] process on the server.\n\nIs fetching over SSH exactly the same as fetching over git:// protocol?\n\n>\n> Then the server will tell the client which SHAs it has for each ref,\n> and the client figures out what it needs and responds with a list of\n> SHAs it wants and already has.\n>\n> At this point, the server will generate a packfile with all the\n> objects that the client needs and begin streaming it down to the\n> client.\n\nWe would want here probably the overview of client-server communication\nas described in Documentation/technical/pack-protocol.txt\n\n>\n> Let's look at an example.\n>\n> The client connects and sends the request header. The clone command\n>\n> \t$ git clone git://myserver.com/project.git\n>\n> produces the following request:\n>\n> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n>\n> The first four bytes contain the hex length of the line (including 4\n> byte line length and trailing newline if present). Following are the\n> command and arguments. This is followed by a null byte and then the\n> host information. The request is terminated by a null byte.\n\nThere is a question how to organize this information. Should we describe\npkt-line format upfront, e.g. using ABNF notation from RFC 5234 used in\nRFC documents:\n\n = ( [ LF ] ) / \n = 4HEXDIGIT ; length of \n = \"0000\"\n\nor something like that?\n\n\nSidenote: wouldn't it be better to use \\0 (\\\\0 in source) for NUL\ncharacter rather than \\000 (\\\\000 in source) octal representation?\n\n>\n> The request is processed and turned into a call to git-upload-pack:\n>\n> \t$ git-upload-pack /path/to/repos/project.git\n\nIs it \"git-upload-pack\" or \"git upload-pack\" nowadays?\n\nAdditionally currently this chapter does not explain how request for\n\"/project.git\" is turned into /path/to/repos/project.git path to\nrepository both in case of git-daemon (git:// protocol) and SSH.\n\n>\n> This immediately returns information of the repo:\n\nTo be more exact this is information about references (I guess this\nis information about heads only, is it?), with information about\nserver capabilities stuffed in.\n\n>\n> \t007c74730d410fcb6603ace96f1dc55ea6196122532d HEAD\\\\000multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress\n> 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n> 003d5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/dist\n> 003e7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 refs/heads/local\n> 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n> 0000 \n>\n> Each line starts with a four byte line length declaration in hex. The\n> section is terminated by a line length declaration of 0000.\n\nShould we describe here, or in appendix, or in sidenote, or in footnote\nall currently supported client capabilities and server capabilities?\n\n * multi_ack (why not mult-ack?)\n * thin-pack \n * side-band \n * side-band-64k \n * ofs-delta \n * shallow \n * no-progress\n\nIs each line terminated by \"\\n\" or \"\\0\"? Is 'flush' line? This is not\nclear from above description. From simple playing with nc (netcat) it\nlooks like each line with exception of '0000' is terminated with \"\\n\".\n\n>\n> This is sent back to the client verbatim. The client responds with\n> another request:\n>\n> \t0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta \n> \t0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n> \t0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n> \t0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n> \t0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n\nThe semantics (meaning) of those 'want' lines is not described here,\nalthough one can easily guess that those are commits that client does\nnot have, and which do want. In the case of \"git clone\" those are all\nunique sha1 that client got (what happend if server has detached HEAD?)\n\nIt is not clear, but one can guess that set of capabilities that client\nsends (without stuffing behind NUL character this time?) is a supported\nby client and wanted subset of server capabilities.\n\n> \t00000009done\n\nFirst I thought that this is an error... but not, the 'flush' (\"0000\")\nis not LF terminated.\n\n>\n> The is sent to the open git-upload-pack process which then streams\n> out the final response:\n\nHmmm... here it is used different notation than above; everything is\nwithin quotes, and end-of-line character is explicitly stated this time.\n\n>\n> \t\"0008NAK\\n\"\n\nWhat does this server response mean? That served doesn't need more\ninfo? Having overview of client-server communication upfront would help\nhere (there would be a point to refer to).\n\n> \t\"0023\\\\002Counting objects: 2797, done.\\n\"\n> \t\"002b\\\\002Compressing objects: 0% (1/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 1% (12/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 2% (24/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 3% (36/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 4% (48/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 5% (59/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 6% (71/1177) \\r\"\n> \t\"0053\\\\002Compressing objects: 7% (83/1177) \\rCompressing objects: 8% (95/1177) \\r\" ...\n> \t\"005b\\\\002Compressing objects: 100% (1177/1177) \\rCompressing objects: 100% (1177/1177), done.\\n\"\n\nI guess that it is sideband support: after pkt-length there is number\nof stream (multiplexing), where 2 = \\002 means stderr.\n\nI wonder why sometimes it is one line per update, and sometimes there\nis more than one update info stuffed in single line.\n\n> \"2004\\\\001PACK\\\\000\\\\000\\\\000\\\\002\\\\000\\\\000\\n\\\\355\\\\225\\\\017x\\\\234\\\\235\\\\216K\\n\\\\302\"...\n> \"2005\\\\001\\\\360\\\\204{\\\\225\\\\376\\\\330\\\\345]z\\226\\273\"...\n> \t...\n> \t\"0037\\\\002Total 2797 (delta 1799), reused 2360 (delta 1529)\\n\"\n> \t...\n\nI can guess that this is example of multiplexing at work. Here again\nsome kind of ABNF notation would be IMHO useful, e.g.\n\n = [ LF / CR ]\n = 4HEXDIGIT ; length of \n = %d01-%d02\n\n(Or something like that; I am not sure about ABNF details here).\n\n> \t\"<\\\\276\\\\255L\\\\273s\\\\005\\\\001w0006\\\\001[0000\"\n\nHmmm... strange, this is not in pkt-line format...\n\n>\n> See the Packfile chapter previously for the actual format of the\n> packfile data in the response.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115294","messageId":"20090602232724.GN30527@spearce.org","threadId":"19323","inReplyTo":"200906022339.08639.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-02T23:27:24Z","receivedAt":"2009-06-02T23:27:24Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> I see that there is (at least beginnings of) description of git pack\n> protocol in section \"Transfer Protocols\"[1][2] of chapter \"7. Internals\n> and Plumbing\" of \"Git Community Book\".\n> \n> [1] http://book.git-scm.com/7_transfer_protocols.html\n> [2] http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\n> \n> > ### Fetching Data with Upload Pack ###\n> >\n> > For the smarter protocols, fetching objects is much more efficient. \n> > A socket is opened, either over ssh or over port 9418 (in the case of\n> > the git:// protocol), and the linkgit:git-fetch-pack[1] command on\n> > the client begins communicating with a forked\n> > linkgit:git-upload-pack[1] process on the server.\n> \n> Is fetching over SSH exactly the same as fetching over git:// protocol?\n\nYes. Except git:// starts off by sending \"git-receive-pack\n'repo.git'\" on the wire using a pkt-line format, while ssh:// sends\nthat by way of the remote exec support built into the SSH protocol.\nIOW, the only way that git:// differs from SSH is by providing the\nsmallest shim possible to replace that remote exec feature.\n \n> > Let's look at an example.\n> >\n> > The client connects and sends the request header. The clone command\n> >\n> > \t$ git clone git://myserver.com/project.git\n> >\n> > produces the following request:\n> >\n> > \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n> >\n> > The first four bytes contain the hex length of the line (including 4\n> > byte line length and trailing newline if present). Following are the\n> > command and arguments. This is followed by a null byte and then the\n> > host information. The request is terminated by a null byte.\n> \n> There is a question how to organize this information. Should we describe\n> pkt-line format upfront, e.g. using ABNF notation from RFC 5234 used in\n> RFC documents:\n> \n> = ( [ LF ] ) / \n> = 4HEXDIGIT ; length of \n> = \"0000\"\n> \n> or something like that?\n\nYes.\n \n> Sidenote: wouldn't it be better to use \\0 (\\\\0 in source) for NUL\n> character rather than \\000 (\\\\000 in source) octal representation?\n\nMost languages today honor '\\0' or \"\\0\" as a means of embedding a\nNUL into a char type. So \\0 seems correct to me.\n\n> > The request is processed and turned into a call to git-upload-pack:\n> >\n> > \t$ git-upload-pack /path/to/repos/project.git\n> \n> Is it \"git-upload-pack\" or \"git upload-pack\" nowadays?\n\nSadly, we still invoke \"git-upload-pack\" IIRC.\n \n> > This immediately returns information of the repo:\n> \n> To be more exact this is information about references (I guess this\n> is information about heads only, is it?)\n\nNo, its *all* refs. `git for-each-ref` plus HEAD.\n\n> , with information about\n> server capabilities stuffed in.\n> \n> >\n> > \t007c74730d410fcb6603ace96f1dc55ea6196122532d HEAD\\\\000multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress\n> > 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n> > 003d5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/dist\n> > 003e7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 refs/heads/local\n> > 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n> > 0000 \n> >\n> > Each line starts with a four byte line length declaration in hex. The\n> > section is terminated by a line length declaration of 0000.\n> \n> Should we describe here, or in appendix, or in sidenote, or in footnote\n> all currently supported client capabilities and server capabilities?\n\nYes.\n\n> * multi_ack (why not mult-ack?)\n\nHysterical rasins. ;-)\n\n> * thin-pack \n> * side-band \n> * side-band-64k \n> * ofs-delta \n> * shallow \n> * no-progress\n> \n> Is each line terminated by \"\\n\" or \"\\0\"?\n\nActually, its weird... Each line is terminated by a \"\\n\" by\nconvention only, which is included in the 4 byte length declaration.\nAfter reading a line the client slaps a NUL onto the end at the\nposition indicated by the length declaration, and processes the\nline, skipping the \"\\n\" at the end if it is present, and sliently\naccepting the line just fine if the \"\\n\" is missing.\n\nThis is why the \"\\0capability\" hack works, the client didn't care\nthat that first ref doesn't end in an LF. But it stopped where that\n\"\\0\" was because it was using a C string style operator.\n\n> Is 'flush' line? This is not\n> clear from above description. From simple playing with nc (netcat) it\n> looks like each line with exception of '0000' is terminated with \"\\n\".\n\nThe only reason we end with \"\\n\" is to make playing with netcat\neasier. There isn't a real practical reason in terms of the protocol\nfor why you need that \"\\n\" in there.\n\nBut. That flush line is magical. A length of \"0000\" means its a\nflush packet, which has no data payload. An \"\\n\" after the \"0000\"\nwould break the protocol as the server would read that \"\\n\" in a\ncontext where it is expecting another pkt-line length declaration.\n\"\\n\" is not a hex digit, so \"0000\\n\" is horribly horribly broken.\n \n> > This is sent back to the client verbatim. The client responds with\n> > another request:\n> >\n> > \t0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta \n> > \t0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n> > \t0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n> > \t0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n> > \t0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n> \n> The semantics (meaning) of those 'want' lines is not described here,\n> although one can easily guess that those are commits that client does\n> not have, and which do want. In the case of \"git clone\" those are all\n> unique sha1 that client got (what happend if server has detached HEAD?)\n\nIIRC, HEAD isn't fetched if its detached.\n\nThe client pattern matches the advertisements against the fetch\nrefspec, which is \"refs/heads/*:refs/remotes/origin/*\" by default.\nHEAD doesn't match the LHS, so it doesn't get wanted by the client.\n \n> It is not clear, but one can guess that set of capabilities that client\n> sends (without stuffing behind NUL character this time?) is a supported\n> by client and wanted subset of server capabilities.\n\nYes. Another oddity. Why the heck we didn't also use the NUL hack\nhere is a good question. Basically, the NUL hack wasn't necessary\nin the server at the time that capabilities were added, because the\nserver was parsing the line with a fixed position parser. It only\nlooked at the first 45 characters (\"want 0x40\"). Anything after\nthat was assumed to be garbage... like that unnecessary LF.\n \n> > \t00000009done\n> \n> First I thought that this is an error... but not, the 'flush' (\"0000\")\n> is not LF terminated.\n\nCorrect. Again, server only cares that its \"done\" in a packet.\nI think \"donedammitsendmeapacknow\" is also going to make the current\nservers spit back a pack. :-)\n \n> >\n> > \t\"0008NAK\\n\"\n> \n> What does this server response mean? That served doesn't need more\n> info?\n\nIt means the server is answering a prior flush from the client,\nand is saying \"I still can't serve you, keep tell me more have\".\n\n> > \t\"0023\\\\002Counting objects: 2797, done.\\n\"\n> > \t\"002b\\\\002Compressing objects: 0% (1/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 1% (12/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 2% (24/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 3% (36/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 4% (48/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 5% (59/1177) \\r\"\n> > \t\"002c\\\\002Compressing objects: 6% (71/1177) \\r\"\n> > \t\"0053\\\\002Compressing objects: 7% (83/1177) \\rCompressing objects: 8% (95/1177) \\r\" ...\n> > \t\"005b\\\\002Compressing objects: 100% (1177/1177) \\rCompressing objects: 100% (1177/1177), done.\\n\"\n> \n> I guess that it is sideband support: after pkt-length there is number\n> of stream (multiplexing), where 2 = \\002 means stderr.\n\nYes. Actually, 2 means \"progress messages, most likely suitable\nfor stderr\". 1 means \"pack data\". 3 means \"fatal error message,\nand we're dead now\".\n \n> I wonder why sometimes it is one line per update, and sometimes there\n> is more than one update info stuffed in single line.\n\nBuffering. There are two processes running on the server side,\ngit-pack-objects is producing these messages on its stderr,\nand the pack data on stdout. Both are actually a pipe read by\ngit-upload-pack in a select loop. If pack-objects can write two\nmessages into the pipe buffer before upload-pack is woken to read\nthem out, upload-pack might find two (or more) messages ready to\nread without blocking. These get bundled into a single packet,\nbecause, why not, its easier to code it that way.\n\nIts most common on the end like that, where we dump 100%, and\nthen immediately add the \", done\" and start a new progress meter.\nIts less likely in the middle, where we try to space out the progress\nupdates to around 1 per second, or 1 per percentage unit.\n \n-- \nShawn.\n"},{"id":"115299","messageId":"200906030250.01413.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090602232724.GN30527@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T00:50:00Z","receivedAt":"2009-06-03T00:50:00Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Thank you very much for your comments!\n\nOn Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n\n>> I see that there is (at least beginnings of) description of git pack\n>> protocol in section \"Transfer Protocols\"[1][2] of chapter \"7. Internals\n>> and Plumbing\" of \"Git Community Book\".\n>> \n>> [1] http://book.git-scm.com/7_transfer_protocols.html\n>> [2] http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\n>> \n>>> ### Fetching Data with Upload Pack ###\n>>>\n>>> For the smarter protocols, fetching objects is much more efficient. \n>>> A socket is opened, either over ssh or over port 9418 (in the case of\n>>> the git:// protocol), and the linkgit:git-fetch-pack[1] command on\n>>> the client begins communicating with a forked\n>>> linkgit:git-upload-pack[1] process on the server.\n>> \n>> Is fetching over SSH exactly the same as fetching over git:// protocol?\n> \n> Yes. Except git:// starts off by sending \"git-receive-pack\n> 'repo.git'\" on the wire using a pkt-line format, while ssh:// sends\n> that by way of the remote exec support built into the SSH protocol.\n> IOW, the only way that git:// differs from SSH is by providing the\n> smallest shim possible to replace that remote exec feature.\n> \n>>> Let's look at an example.\n>>>\n>>> The client connects and sends the request header. The clone command\n>>>\n>>> \t$ git clone git://myserver.com/project.git\n>>>\n>>> produces the following request:\n>>>\n>>> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n[...]\n\nSo this mean that when cloning via SSH \n\n $ git clone ssh://myserver.com/project.git\n\ninstead of this first request git would simply invoke [something like]:\n\n # ssh myserver.com git-upload-pack project.git\n\nisn't it? (I am not sure if it uses \"project.git\" or \"/project.git\", \nand how it does generate full pathname for repository).\n\n\nBTW I wonder why we use stuffing here using \"\\0\" / NUL as separator\ntrick, and whether line has to be terminated with \"\\0\", or can it be\nterminated with \"\\n\".\n\n>> \n>> There is a question how to organize this information. Should we describe\n>> pkt-line format upfront, e.g. using ABNF notation from RFC 5234 used in\n>> RFC documents:\n>> \n>> = ( [ LF ] ) / \n>> = 4HEXDIGIT ; length of \n>> = \"0000\"\n>> \n>> or something like that?\n> \n> Yes.\n\nwhere\n\n HEXDIGIT = 0-9 / a-f\n\nWell, it should probably be spelled in full. Probably, because I have\nno experience with using ABNF... and didn't do my research :-)\n\n HEXDIGIT = DIGIT / \"a\" / \"b\" / \"c\" / \"d\" / \"e\" / \"f\"\n DIGIT = \"0\" / \"1\" / \"2\" / \"3\" / \"4\" / \"5\" / \"6\" / \"7\" / \"8\" / \"9\"\n\n(should HEXDIGIT use lowercase a-f, or can it use uppercase A-F?)\n\n>> Sidenote: wouldn't it be better to use \\0 (\\\\0 in source) for NUL\n>> character rather than \\000 (\\\\000 in source) octal representation?\n> \n> Most languages today honor '\\0' or \"\\0\" as a means of embedding a\n> NUL into a char type. So \\0 seems correct to me.\n\nThat was more a question to Scott Chacon, sorry.\n\nDo Ruby understand \"\\0\", or do you need to spell it \"\\000\"?\n\n> \n>>> The request is processed and turned into a call to git-upload-pack:\n>>>\n>>> \t$ git-upload-pack /path/to/repos/project.git\n>> \n>> Is it \"git-upload-pack\" or \"git upload-pack\" nowadays?\n> \n> Sadly, we still invoke \"git-upload-pack\" IIRC.\n\nSo that is why git-upload-pack has to be in $PATH, or is it only because\nnew server can be used with old clients (before git-cmd moving outside\n$PATH)?\n\n> \n>>> This immediately returns information of the repo:\n>> \n>> To be more exact this is information about references (I guess this\n>> is information about heads only, is it?)\n> \n> No, its *all* refs. `git for-each-ref` plus HEAD.\n\nYou meant probably `git show-ref` plus HEAD, isn't it? \n`git for-each-ref` has different default output...\n\nStill, example should IMHO include at least one tag...\n\n> \n>> , with information about\n>> server capabilities stuffed in.\n>> \n>>>\n>>> \t 007c74730d410fcb6603ace96f1dc55ea6196122532d HEAD\\\\000multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress\n>>> 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n>>> 003d5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/dist\n>>> 003e7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 refs/heads/local\n>>> 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n>>> 0000 \n>>>\n>>> Each line starts with a four byte line length declaration in hex. The\n>>> section is terminated by a line length declaration of 0000.\n>> \n>> Should we describe here, or in appendix, or in sidenote, or in footnote\n>> all currently supported client capabilities and server capabilities?\n> \n> Yes.\n> \n>> * multi_ack (why not multi-ack?)\n> \n> Hysterical rasins. ;-)\n\nWhat does multi_ack capability mean?\n \n>> * thin-pack\n\nServer can send thin packs, i.e. packs which do not contain base \nelements, if those base elements are available on clients side.\nClient has thin-pack capability when it understand how to \"thicken\"\nthem adding required delta bases making them independent.\n\nOf course it doesn't make sense for client to use (request) this\ncapability for git-clone.\n \n>> * side-band \n>> * side-band-64k \n\nThis probably means that server can send, and client understand \nmultiplexed (muxed) progress reports and error info interleaved\nwith the packfile itself.\n\nBut I don't know what is the difference, whether server can provide\nside-band-64k without the other (side-band), and whether client has\nto request only one of those two capabilities.\n\n>> * ofs-delta \n\nServer can send, and client understand PACKv2 with delta refering to\nits base by position in pack rather than by SHA-1... do I understand\nthis correctly?\n\n>> * shallow \n\nServer can send shallow clone (git clone --depth ...).\n\n>> * no-progress\n\nWhat that does mean?\n\n>> \n>> Is each line terminated by \"\\n\" or \"\\0\"?\n> \n> Actually, its weird... Each line is terminated by a \"\\n\" by\n> convention only, which is included in the 4 byte length declaration.\n> After reading a line the client slaps a NUL onto the end at the\n> position indicated by the length declaration, and processes the\n> line, skipping the \"\\n\" at the end if it is present, and sliently\n> accepting the line just fine if the \"\\n\" is missing.\n\nThis probably should be described... \n\nDoes git require that each line is terminated by something (e.g. \"\\n\"),\nor does it not?\n\n> \n> This is why the \"\\0capability\" hack works, the client didn't care\n> that that first ref doesn't end in an LF. But it stopped where that\n> \"\\0\" was because it was using a C string style operator.\n\nIt is a bit pity tat git protocol was not created with extendability\n(like capabilities) in mind...\n\n> \n>> Is 'flush' line? This is not\n>> clear from above description. From simple playing with nc (netcat) it\n>> looks like each line with exception of '0000' is terminated with \"\\n\".\n> \n> The only reason we end with \"\\n\" is to make playing with netcat\n> easier. There isn't a real practical reason in terms of the protocol\n> for why you need that \"\\n\" in there.\n> \n> But. That flush line is magical. A length of \"0000\" means its a\n> flush packet, which has no data payload. An \"\\n\" after the \"0000\"\n> would break the protocol as the server would read that \"\\n\" in a\n> context where it is expecting another pkt-line length declaration.\n> \"\\n\" is not a hex digit, so \"0000\\n\" is horribly horribly broken.\n\nO.K.\n\nIt would probably be more clean to explicitly include terminators in\nthe output (like for final response: sending packfile) and put magical\n'flush' line in separate row (separate line of example output).\n\nBTW. do \"0001\" - \"0003\" pkt-lines are reserved, or just invalid?\n\n> \n>>> This is sent back to the client verbatim. The client responds with\n>>> another request:\n>>>\n>>> \t0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta \n>>> \t0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n>>> \t0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n>>> \t0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n>>> \t0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n>> \n>> The semantics (meaning) of those 'want' lines is not described here,\n>> although one can easily guess that those are commits that client does\n>> not have, and which do want. In the case of \"git clone\" those are all\n>> unique sha1 that client got (what happend if server has detached HEAD?)\n> \n> IIRC, HEAD isn't fetched if its detached.\n> \n> The client pattern matches the advertisements against the fetch\n> refspec, which is \"refs/heads/*:refs/remotes/origin/*\" by default.\n> HEAD doesn't match the LHS, so it doesn't get wanted by the client.\n\nWell, unless client requests 'mirror' clone, where from what \nI understand \"refs/*:refs/*\" is implied... but that would also\nnot match HEAD. Hmmm... I would think that \"git clone --mirror\"\nwould also mirror HEAD if it is detached.\n\n> \n>> It is not clear, but one can guess that set of capabilities that client\n>> sends (without stuffing behind NUL character this time?) is a supported\n>> by client and wanted subset of server capabilities.\n> \n> Yes. Another oddity. Why the heck we didn't also use the NUL hack\n> here is a good question. Basically, the NUL hack wasn't necessary\n> in the server at the time that capabilities were added, because the\n> server was parsing the line with a fixed position parser. It only\n> looked at the first 45 characters (\"want 0x40\"). Anything after\n> that was assumed to be garbage... like that unnecessary LF.\n\nAnd it was necessary when sending server capabilities because that\nresponse cannot be parsed using fixed position parser: HEAD can not\nexists, and refs have arbitrary (well, up to something less than\nPATH_MAX for sure) length.\n\n> \n>>> \t00000009done\n>> \n>> First I thought that this is an error... but not, the 'flush' (\"0000\")\n>> is not LF terminated.\n> \n> Correct. Again, server only cares that its \"done\" in a packet.\n> I think \"donedammitsendmeapacknow\" is also going to make the current\n> servers spit back a pack. :-)\n> \n>>>\n>>> \t\"0008NAK\\n\"\n>> \n>> What does this server response mean? That served doesn't need more\n>> info?\n> \n> It means the server is answering a prior flush from the client,\n> and is saying \"I still can't serve you, keep tell me more have\".\n\nHmmm... the communication between server and client is not entirely\nclean. Do I understand correctly that this NAK is response to clients\nflush after all those \"want\" lines? And that \"0009done\" from client\ntells server that it should send everything it has?\n\n> \n>>> \t\"0023\\\\002Counting objects: 2797, done.\\n\"\n>>> \t\"002b\\\\002Compressing objects: 0% (1/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 1% (12/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 2% (24/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 3% (36/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 4% (48/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 5% (59/1177) \\r\"\n>>> \t\"002c\\\\002Compressing objects: 6% (71/1177) \\r\"\n>>> \t\"0053\\\\002Compressing objects: 7% (83/1177) \\rCompressing objects: 8% (95/1177) \\r\" ...\n>>> \t\"005b\\\\002Compressing objects: 100% (1177/1177) \\rCompressing objects: 100% (1177/1177), done.\\n\"\n>> \n>> I guess that it is sideband support: after pkt-length there is number\n>> of stream (multiplexing), where 2 = \\002 means stderr.\n> \n> Yes. Actually, 2 means \"progress messages, most likely suitable\n> for stderr\". 1 means \"pack data\". 3 means \"fatal error message,\n> and we're dead now\".\n\nBut it is easily extendable, i.e. sideband > 3 would work, although\nbe ignored, isn't it?\n\nBy the way, how client does know that server started to send final\ndata, i.e. packfile multiplexed / interleaved with progress reports,\nand should expect rather than output?\n\n> \n>> I wonder why sometimes it is one line per update, and sometimes there\n>> is more than one update info stuffed in single line.\n> \n> Buffering. There are two processes running on the server side,\n> git-pack-objects is producing these messages on its stderr,\n> and the pack data on stdout. Both are actually a pipe read by\n> git-upload-pack in a select loop. If pack-objects can write two\n> messages into the pipe buffer before upload-pack is woken to read\n> them out, upload-pack might find two (or more) messages ready to\n> read without blocking. These get bundled into a single packet,\n> because, why not, its easier to code it that way.\n> \n> Its most common on the end like that, where we dump 100%, and\n> then immediately add the \", done\" and start a new progress meter.\n> Its less likely in the middle, where we try to space out the progress\n> updates to around 1 per second, or 1 per percentage unit.\n\nAhhh... now I understand. Thanks.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115303","messageId":"20090603012940.GA3355@spearce.org","threadId":"19323","inReplyTo":"200906030250.01413.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T01:29:40Z","receivedAt":"2009-06-03T01:29:40Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> >>>\n> >>> The client connects and sends the request header. The clone command\n> >>>\n> >>> \t$ git clone git://myserver.com/project.git\n> >>>\n> >>> produces the following request:\n> >>>\n> >>> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n> [...]\n> \n> So this mean that when cloning via SSH \n> \n> $ git clone ssh://myserver.com/project.git\n> \n> instead of this first request git would simply invoke [something like]:\n> \n> # ssh myserver.com git-upload-pack project.git\n\nActually, \n\n # ssh myserver.com git-upload-pack /project.git\n \n> isn't it? (I am not sure if it uses \"project.git\" or \"/project.git\", \n> and how it does generate full pathname for repository).\n\nIn an ssh:// format URI, its absolute in the URI, so the / after\nthe host name (or port number) is sent as an argument, which is then\nread by the remote git-upload-pack exactly as is, so its effectively\nan absolute path in the remote filesystem.\n\nIn a \"user@host:path\" format URI, its relative to the user's home\ndirectory, because we run:\n\n # ssh user@host git-upload-pack path\n\n> BTW I wonder why we use stuffing here using \"\\0\" / NUL as separator\n> trick, and whether line has to be terminated with \"\\0\", or can it be\n> terminated with \"\\n\".\n\nStuffing here? What are we talking about again?\n \n> >>> The request is processed and turned into a call to git-upload-pack:\n> >>>\n> >>> \t$ git-upload-pack /path/to/repos/project.git\n> >> \n> >> Is it \"git-upload-pack\" or \"git upload-pack\" nowadays?\n> > \n> > Sadly, we still invoke \"git-upload-pack\" IIRC.\n> \n> So that is why git-upload-pack has to be in $PATH, or is it only because\n> new server can be used with old clients (before git-cmd moving outside\n> $PATH)?\n\nBoth. :-)\n\nClients (old and new) ask for git-upload-pack and git-receive-pack,\nby SSH as we saw above... so it needs to be in the remote $PATH.\n \n> >>> This immediately returns information of the repo:\n> >> \n> >> To be more exact this is information about references (I guess this\n> >> is information about heads only, is it?)\n> > \n> > No, its *all* refs. `git for-each-ref` plus HEAD.\n> \n> You meant probably `git show-ref` plus HEAD, isn't it? \n> `git for-each-ref` has different default output...\n\nWhatever. I was talking about what we enumerate, not the output\nformat. The server code is actually not using either of those\nprograms, but is instead just making the direct calls to refs.c\nfunctions (for_each_ref function) and formatting the result specially\nfor the protocol.\n \n> > Hysterical rasins. ;-)\n> \n> What does multi_ack capability mean?\n\nIt allows the server to return \"ACK $SHA1 continue\" as soon as\nit finds a commit that it can use as a common base, between the\nclient's wants and the client's have set.\n\nBy sending this early, the server can potentially head off the\nclient from walking any further down that particular branch of the\nclient's repository history. The client may still need to walk down\nother branches, sending have lines for those, until the server has\na complete cut across the DAG, or the client has said \"done\".\n\nIIRC, without multi_ack, a client sends have lines in --date-order\nuntil the server has found a common base. That means the client\nwill send have lines that are already known by the server to be\ncommon, because they overlap in time with another branch that the\nserver hasn't found a common base on yet.\n\nE.g. the client has things in caps that the server doesn't; server\nhas things in lower case:\n\n +---- u ---------------------- x\n / +----- y\n / /\n a -- b -- c -- d -- E -- F\n \\\n +--- Q -- R -- S\n\nIf the client wants x,y and starts out by saying have F,S, the\nserver doesn't know what F,S is. Eventually the client says \"have\nd\" and the server sends \"ACK d continue\" to let the client know to\nstop walking down that line (so don't send c-b-a), but its not done\nyet, it needs a base for X. The client keeps going with S-R-Q,\nuntil a gets reached, at which point the server has a clear base\nand it all ends.\n\nWithout multi_ack the client would have sent that c-b-a chain anyway,\ninterleaved with S-R-Q.\n\nJunio, am I right? I think I am, but I've had to reverse engineer\nmost of this. And the above is my understanding of it.\n\n> >> * thin-pack\n> \n> Server can send thin packs, i.e. packs which do not contain base \n> elements, if those base elements are available on clients side.\n> Client has thin-pack capability when it understand how to \"thicken\"\n> them adding required delta bases making them independent.\n\nYes.\n \n> Of course it doesn't make sense for client to use (request) this\n> capability for git-clone.\n\nNo, no it doesn't. But if the client does request it (and I think\nmodern clients actually do request it, even on initial clone case)\nthe server won't produce a thin pack. Why? There is no common base,\nso there is no uninteresting set to omit from the pack. :-)\n\n> >> * side-band \n> >> * side-band-64k \n> \n> This probably means that server can send, and client understand \n> multiplexed (muxed) progress reports and error info interleaved\n> with the packfile itself.\n> \n> But I don't know what is the difference, whether server can provide\n> side-band-64k without the other (side-band), and whether client has\n> to request only one of those two capabilities.\n\nYes. These two options are mutually exclusive. A client should\nask for only one of them, and a modern client always favors\nside-band-64k.\n\nLong ago, we only had side-band, which allowed up to 1000 bytes\nper packet. But the packet length field is 4 bytes, in hex, so 16\nbits worth of information space. Limiting it to only 1000 bytes\nfor a large 800 MiB binary pack file on initial clone is really\nquite poor usage of the data stream space.\n\nWe couldn't just up the limit the server sends to the full 2^16\nbecause older clients literally had a char[1000] allocated on the\nstack, and we'd overflow it. So \"side-band-64k\" came about as\nanother way for the client to request side-band, but to also say\nit can handle the much larger packets, packets that are actually\ncrammed nearly full (65520 bytes).\n \n> >> * ofs-delta \n> \n> Server can send, and client understand PACKv2 with delta refering to\n> its base by position in pack rather than by SHA-1... do I understand\n> this correctly?\n\nYes. Its that they can send/read OBJ_OFS_DELTA, aka type 6 in\na pack file.\n \n> >> * shallow \n> \n> Server can send shallow clone (git clone --depth ...).\n> \n> >> * no-progress\n> \n> What that does mean?\n\nThe client was started with \"git clone -q\" or something, and doesn't\nwant that side brand 2. Basically the client just says \"I do not\nwish to receive stream 2 on sideband, so do not send it to me,\nand if you did, I will drop it on the floor anyway\".\n \n> >> Is each line terminated by \"\\n\" or \"\\0\"?\n> > \n> > Actually, its weird... Each line is terminated by a \"\\n\" by\n> > convention only, which is included in the 4 byte length declaration.\n> > After reading a line the client slaps a NUL onto the end at the\n> > position indicated by the length declaration, and processes the\n> > line, skipping the \"\\n\" at the end if it is present, and sliently\n> > accepting the line just fine if the \"\\n\" is missing.\n> \n> This probably should be described... \n> \n> Does git require that each line is terminated by something (e.g. \"\\n\"),\n> or does it not?\n\nIt doesn't, but convention says to include the \"\\n\" to be nice to\na human. Junio may argue that it does require it, I don't know,\nbut if you read through any modern implementation (e.g. C Git or\nJGit) the \"\\n\" is entirely optional when parsing the lines.\n\nActually, it has to be, because that \"\\n\" isn't there on the first\nline when the capability data appears either wedged behind the \"\\0\"\nor after the \" \" at the end of the line.\n \n> > This is why the \"\\0capability\" hack works, the client didn't care\n> > that that first ref doesn't end in an LF. But it stopped where that\n> > \"\\0\" was because it was using a C string style operator.\n> \n> It is a bit pity tat git protocol was not created with extendability\n> (like capabilities) in mind...\n\nYes, no doubt. There are many things I would have done differently,\ngiven that I now have 20/20 hindsight vision into the past's future.\n\n:-)\n\nThe protocol (mostly) works fine as-is. Its widely distributed in\nterms of clients using it on a daily basis. Its likely to continue\nto serve our needs well into the future. So, it is what it is.\n \n> BTW. do \"0001\" - \"0003\" pkt-lines are reserved, or just invalid?\n\nInvalid. No clue if they are considered \"reserved for future use\".\nI don't think they are, I think they just out flat out not something\nany client can ever sanely produce.\n\nBut, hey, look, another back door we can use in the future to\nwedge something else into this protocol, after introducing a new\ncapability for it. :-)\n \n> >>>\n> >>> \t\"0008NAK\\n\"\n> >> \n> >> What does this server response mean? That served doesn't need more\n> >> info?\n> > \n> > It means the server is answering a prior flush from the client,\n> > and is saying \"I still can't serve you, keep tell me more have\".\n> \n> Hmmm... the communication between server and client is not entirely\n> clean. Do I understand correctly that this NAK is response to clients\n> flush after all those \"want\" lines?\n\nYes.\n\n> And that \"0009done\" from client\n> tells server that it should send everything it has?\n\nYes. It means the client will not issue any more \"have\" lines,\nas it has nothing further in its history, so the server just has\nto give up and start generating a pack based on what it knows.\n\n> > Yes. Actually, 2 means \"progress messages, most likely suitable\n> > for stderr\". 1 means \"pack data\". 3 means \"fatal error message,\n> > and we're dead now\".\n> \n> But it is easily extendable, i.e. sideband > 3 would work, although\n> be ignored, isn't it?\n\nCorrect.\n \n> By the way, how client does know that server started to send final\n> data, i.e. packfile multiplexed / interleaved with progress reports,\n> and should expect rather than output?\n\nAfter the client receives a \"ACK\" or \"NAK\" for the number of\noutstanding flushes it still has, *after* it has sent \"done\".\nThis also varies based on whether or not multi_ack was enabled.\n\nIts ugly. But basically you keep a running counter of each \"flush\"\nsent, and then you send a \"done\" out, and then you wait until\nyou have the right number of ACK/NAK answers back, and then the\nstream changes format.\n \n-- \nShawn.\n"},{"id":"115307","messageId":"7vab4qdq2j.fsf@alter.siamese.dyndns.org","threadId":"19323","inReplyTo":"20090603012940.GA3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-06-03T02:11:16Z","receivedAt":"2009-06-03T02:11:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> IIRC, without multi_ack, a client sends have lines in --date-order\n> until the server has found a common base. That means the client\n> will send have lines that are already known by the server to be\n> common, because they overlap in time with another branch that the\n> server hasn't found a common base on yet.\n>\n> E.g. the client has things in caps that the server doesn't; server\n> has things in lower case:\n>\n> +---- u ---------------------- x\n> / +----- y\n> / /\n> a -- b -- c -- d -- E -- F\n> \\\n> +--- Q -- R -- S\n>\n> If the client wants x,y and starts out by saying have F,S, the\n> server doesn't know what F,S is. Eventually the client says \"have\n> d\" and the server sends \"ACK d continue\" to let the client know to\n> stop walking down that line (so don't send c-b-a), but its not done\n> yet, it needs a base for X. The client keeps going with S-R-Q,\n> until a gets reached, at which point the server has a clear base\n> and it all ends.\n>\n> Without multi_ack the client would have sent that c-b-a chain anyway,\n> interleaved with S-R-Q.\n>\n> Junio, am I right?\n\nCorrect.\n\n>> >> * thin-pack\n>> \n>> Server can send thin packs, i.e. packs which do not contain base \n>> elements, if those base elements are available on clients side.\n>> Client has thin-pack capability when it understand how to \"thicken\"\n>> them adding required delta bases making them independent.\n>\n> Yes.\n> \n>> Of course it doesn't make sense for client to use (request) this\n>> capability for git-clone.\n>\n> No, no it doesn't. But if the client does request it (and I think\n> modern clients actually do request it, even on initial clone case)\n> the server won't produce a thin pack. Why? There is no common base,\n> so there is no uninteresting set to omit from the pack. :-)\n\nThere also is \"clone --reference\".\n\n> Actually, it has to be, because that \"\\n\" isn't there on the first\n> line when the capability data appears either wedged behind the \"\\0\"\n> or after the \" \" at the end of the line.\n\nCorrect.\n\n> Its ugly. But basically you keep a running counter of each \"flush\"\n> sent, and then you send a \"done\" out, and then you wait until\n> you have the right number of ACK/NAK answers back, and then the\n> stream changes format.\n\nOne thing that I did not see mentioned in this thread is that the\nimplementation is allowed to buffer non-flush packets and send multiple of\nthem out with a single write(2). In other words, packet_write() could\nbuffer instead of directly calling safe_write(), while packet_flush() must\ndo safe_write() and make sure it drains.\n"},{"id":"115308","messageId":"20090603021518.GB3355@spearce.org","threadId":"19323","inReplyTo":"7vab4qdq2j.fsf@alter.siamese.dyndns.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T02:15:18Z","receivedAt":"2009-06-03T02:15:18Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> One thing that I did not see mentioned in this thread is that the\n> implementation is allowed to buffer non-flush packets and send multiple of\n> them out with a single write(2). In other words, packet_write() could\n> buffer instead of directly calling safe_write(), while packet_flush() must\n> do safe_write() and make sure it drains.\n\nGood point.\n\nThat's one reason why in JGit I call the flush packet of \"0000\"\nend(), and flush() triggers the drain. JGit buffers everything\nits writing, but only by one standard \"have\" window IIRC.\n\nJGit server code triggers a flush() after side-band channel 2 packet\nends, but not an end(), because we only want to drain to the network,\nnot inject a bad \"0000\" packet in the stream.\n\n-- \nShawn.\n"},{"id":"115309","messageId":"20090603021840.GE3042@orbis-terrarum.net","threadId":"19323","inReplyTo":"200906030250.01413.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2009-06-03T02:18:40Z","receivedAt":"2009-06-03T02:18:40Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"> >>> \t\"0008NAK\\n\"\n> >> What does this server response mean? That served doesn't need more\n> >> info?\n> > It means the server is answering a prior flush from the client,\n> > and is saying \"I still can't serve you, keep tell me more have\".\n> Hmmm... the communication between server and client is not entirely\n> clean. Do I understand correctly that this NAK is response to clients\n> flush after all those \"want\" lines? And that \"0009done\" from client\n> tells server that it should send everything it has?\nRelatedly with the \"done\" message, I'm in the process of writing a hook\nthat allows the server to deny the client at this point, instead of\nbuilding and sending a pack.\n\nSuggestions on other modifications that might be needed to integrate. \nThe hook:\n- takes all want/have lines as input (maybe capabilities too?)\n- returns 0/1\n- on error, should also send a message to stderr, to be passed over the\n wire.\n\nMy intended use is to block initial clones while still allowing updates\n(as long as you've got a tree at least commit X recent, I'll talk to\nyou). Initial and too-old clients get a message to go and download a\nbundle instead.\n\n-- \nRobin Hugh Johnson\nGentoo Linux Developer & Infra Guy\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85\n"},{"id":"115326","messageId":"200906031121.38616.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090603012940.GA3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T09:21:37Z","receivedAt":"2009-06-03T09:21:37Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n>> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n>>>>>\n>>>>> The client connects and sends the request header. The clone command\n>>>>>\n>>>>> \t$ git clone git://myserver.com/project.git\n>>>>>\n>>>>> produces the following request:\n>>>>>\n>>>>> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n>> [...]\n>> \n>> So this mean that when cloning via SSH \n>> \n>> $ git clone ssh://myserver.com/project.git\n>> \n>> instead of this first request git would simply invoke [something like]:\n>> \n>> # ssh myserver.com git-upload-pack project.git\n> \n> Actually, \n> \n> # ssh myserver.com git-upload-pack /project.git\n> \n>> isn't it? (I am not sure if it uses \"project.git\" or \"/project.git\", \n>> and how it does generate full pathname for repository).\n> \n> In an ssh:// format URI, its absolute in the URI, so the / after\n> the host name (or port number) is sent as an argument, which is then\n> read by the remote git-upload-pack exactly as is, so its effectively\n> an absolute path in the remote filesystem.\n> \n> In a \"user@host:path\" format URI, its relative to the user's home\n> directory, because we run:\n> \n> # ssh user@host git-upload-pack path\n\nBy the way, this accidentally shows why one might want to prefer \nscp-like / ssh-like \"URL\" for SSH fetch / push, i.e.\n\n [user@]myserver.com:/path/to/repo.git/\n\nrather than ssh:// URL version\n\n ssh://[user@]myserver.com/path/to/repo.git/\n\nOn the other hand I think only URL version allows to specify\nnonstandard port (well, that and ~/.ssh/config).\n\n>> BTW I wonder why we use stuffing here using \"\\0\" / NUL as separator\n>> trick, and whether line has to be terminated with \"\\0\", or can it be\n>> terminated with \"\\n\".\n> \n> Stuffing here? What are we talking about again?\n\nI'm sorry, I was too cryptic here.\n\nI meant that in the request line for fetching via git:// protocol\n\n\t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n\nyou separate path to repository from extra options using \"\\0\" / NUL\nas a separator. Well, this is only sane separator, as it is path \nterminator, the only character which cannot appear in pathname \n(although I do wonder whether project names with e.g. control \ncharacters or UTF-8 characters would work correctly).\n\nIs the final terminating character required to be NUL (\"\\0\"), or can\nit be for LF (\"\\n\"), i.e.\n\n\t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\n\n\nWhat options besides (required?) \"host=[:]\" are supported?\nDo I understand correctly that \"host=\" information is required\nfor core.gitProxy to work, isn't it?\n\n>>>> * no-progress\n>> \n>> What that does mean?\n> \n> The client was started with \"git clone -q\" or something, and doesn't\n> want that side brand 2. Basically the client just says \"I do not\n> wish to receive stream 2 on sideband, so do not send it to me,\n> and if you did, I will drop it on the floor anyway\".\n\nDoes this mean that if server does not support \"no-progress\" capability\nthen client is required to drop diagnostic by itself? Can client request\nto not use sideband (multiplexing) if it is asking for \"no-progress\";\nor is multiplexing required for possible signaling of error condition \non channel 3?\n \n>> It is a bit pity that git protocol was not created with extendability\n>> (like capabilities) in mind...\n> \n> Yes, no doubt. There are many things I would have done differently,\n> given that I now have 20/20 hindsight vision into the past's future.\n> \n> :-)\n> \n> The protocol (mostly) works fine as-is. Its widely distributed in\n> terms of clients using it on a daily basis. Its likely to continue\n> to serve our needs well into the future. So, it is what it is.\n\nI do wonder if existing Internet Standard (in the meaning of RFC) \nprotocols also have such kludges and hacks...\n\n>> By the way, how client does know that server started to send final\n>> data, i.e. packfile multiplexed / interleaved with progress reports,\n>> and should expect rather than output?\n> \n> After the client receives a \"ACK\" or \"NAK\" for the number of\n> outstanding flushes it still has, *after* it has sent \"done\".\n> This also varies based on whether or not multi_ack was enabled.\n> \n> Its ugly. But basically you keep a running counter of each \"flush\"\n> sent, and then you send a \"done\" out, and then you wait until\n> you have the right number of ACK/NAK answers back, and then the\n> stream changes format.\n\nHmmm... perhaps it would be better if pkt-line-sideband had some\ndistinguishing characteristics from ordinary pkt-line, or that sending\nmultiplexed (with sideband) output was preceded by some signal like\n\"0001\" or \"0004\" or \"0005\\n\", or \"000dsideband\\n\". But as you said\nhindsight is 20/20.\n\n\nP.S. By the way, is pkt-line format original invention, or was it \n'borrowed' from some other standard or protocol?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115332","messageId":"m3r5y1a91m.fsf@localhost.localdomain","threadId":"19323","inReplyTo":"20090603021840.GE3042@orbis-terrarum.net","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T10:47:21Z","receivedAt":"2009-06-03T10:47:21Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Please try to not cull the CC list (if possible), and provide\nattribution for quoting.\n\n\"Robin H. Johnson\" writes:\n\n>>>>> \t\"0008NAK\\n\"\n>>>>>\n>>>> What does this server response mean? That served doesn't need more\n>>>> info?\n>>>>\n>>> It means the server is answering a prior flush from the client,\n>>> and is saying \"I still can't serve you, keep tell me more have\".\n>>>\n>> Hmmm... the communication between server and client is not entirely\n>> clean. Do I understand correctly that this NAK is response to clients\n>> flush after all those \"want\" lines? And that \"0009done\" from client\n>> tells server that it should send everything it has?\n\n> Relatedly with the \"done\" message, I'm in the process of writing a hook\n> that allows the server to deny the client at this point, instead of\n> building and sending a pack.\n> \n> Suggestions on other modifications that might be needed to integrate. \n> The hook:\n> - takes all want/have lines as input (maybe capabilities too?)\n> - returns 0/1\n> - on error, should also send a message to stderr, to be passed over the\n> wire.\n\nI am not sure if it would be possible to fit a hook there, but perhaps\nit would be possible to add such `pre-upload` hook... Note that it\nwould have to somehow work for both git:// and ssh:// protocols, and\nperhaps also for \"dumb\" protocols such as http:// (and other\ncurl-based) and deprecated rsync://\n\n> \n> My intended use is to block initial clones while still allowing updates\n> (as long as you've got a tree at least commit X recent, I'll talk to\n> you). Initial and too-old clients get a message to go and download a\n> bundle instead.\n\nWouldn't it be better to make use of mirror-sync (which sadly is in\nplanning stages only; see SoC2009Ideas page on git wiki) to redirect\nto some other repository to be used for cloning requests?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"115339","messageId":"200906031429.28967.jnareb@gmail.com","threadId":"19323","inReplyTo":"200906022339.08639.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T12:29:24Z","receivedAt":"2009-06-03T12:29:24Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 2 Jun 2009, Jakub Narebski wrote:\n> Should we describe here, or in appendix, or in sidenote, or\n> in a footnote, all currently supported client capabilities\n> and server capabilities? \n> \n>  * multi_ack (why not multi-ack?)\n>  * thin-pack \n>  * side-band \n>  * side-band-64k \n>  * ofs-delta \n>  * shallow \n>  * no-progress\n\nThere is also another capability\n\n * include-tag\n\nWhat does it mean? Is it about sending tags if we are sending objects \nthey point to, or is it about sending all tags?\n\n\nP.S. Is hexdigit length case sensitive i.e. 0-9a-f, or is it not\n case sensitive i.e. 0-9a-fA-F?\n-- \nJakub Narebski\nPoland\n"},{"id":"115342","messageId":"20090603141718.GC3355@spearce.org","threadId":"19323","inReplyTo":"m3r5y1a91m.fsf@localhost.localdomain","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T14:17:19Z","receivedAt":"2009-06-03T14:17:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \"Robin H. Johnson\" writes:\n> \n> > Relatedly with the \"done\" message, I'm in the process of writing a hook\n> > that allows the server to deny the client at this point, instead of\n> > building and sending a pack.\n> \n> I am not sure if it would be possible to fit a hook there, but perhaps\n> it would be possible to add such `pre-upload` hook... Note that it\n> would have to somehow work for both git:// and ssh:// protocols, and\n> perhaps also for \"dumb\" protocols such as http:// (and other\n> curl-based) and deprecated rsync://\n\nUh, that hook can't be used on HTTP or rsync. How do you expect the\nHTTP or rsync client to execute a process on the server? It can't,\nthe server isn't git aware. Its only valid on smart protocols.\n \n> > My intended use is to block initial clones while still allowing updates\n> > (as long as you've got a tree at least commit X recent, I'll talk to\n> > you). Initial and too-old clients get a message to go and download a\n> > bundle instead.\n> \n> Wouldn't it be better to make use of mirror-sync (which sadly is in\n> planning stages only; see SoC2009Ideas page on git wiki) to redirect\n> to some other repository to be used for cloning requests?\n\nYes. But mirror-sync isn't here yet, and is a lot more work to\ncreate than hacking upload-pack.c to invoke a hook.\n\n-- \nShawn.\n"},{"id":"115343","messageId":"20090603141947.GD3355@spearce.org","threadId":"19323","inReplyTo":"200906031429.28967.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T14:19:47Z","receivedAt":"2009-06-03T14:19:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Tue, 2 Jun 2009, Jakub Narebski wrote:\n> > Should we describe here, or in appendix, or in sidenote, or\n> > in a footnote, all currently supported client capabilities\n> > and server capabilities? \n> \n> * include-tag\n> \n> What does it mean? Is it about sending tags if we are sending objects \n> they point to,\n\nYes, this.\n\n> or is it about sending all tags?\n\nNo, not this.\n\nIf we pack an object to the client, and a tag points exactly at\nthat object, we pack the tag too. In general this allows a client\nto get all new tags when it fetches a branch, in a single network\nconnection.\n \n> P.S. Is hexdigit length case sensitive i.e. 0-9a-f, or is it not\n> case sensitive i.e. 0-9a-fA-F?\n\nGit parses both a-f/A-F, but prefers to create a-f.\n\n-- \nShawn.\n"},{"id":"115345","messageId":"20090603144837.GE3355@spearce.org","threadId":"19323","inReplyTo":"200906031121.38616.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T14:48:37Z","receivedAt":"2009-06-03T14:48:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> I'm sorry, I was too cryptic here.\n> \n> I meant that in the request line for fetching via git:// protocol\n> \n> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n> \n> you separate path to repository from extra options using \"\\0\" / NUL\n> as a separator. Well, this is only sane separator, as it is path \n> terminator, the only character which cannot appear in pathname \n> (although I do wonder whether project names with e.g. control \n> characters or UTF-8 characters would work correctly).\n\nNo, that isn't the reason '\\0' is used here. But yea, that is true.\n\nThe reason \\0 is used is, git-daemon reads the 4 byte length, decodes\nthat, then reads that many bytes. Finally it writes a '\\0' at the\nend of what it read, so that the entire \"line\" is NUL terminated.\nThen it reads the \"command path\" part from the resulting C string.\n\nThe host=myserver.com part came later, after many daemons were\nalready running all over the world. By hiding it behind the '\\0'\nan old daemon would never see it (but strlen() returned a value that\nwas less than the length read, but the old daemons didn't care).\nNewer daemons look for where strlen() < length, and assume that\nthe host header follows.\n\nThe host header ends with '\\0' in case additional headers would\nalso appear here in the future. IOW, like HTTP allows new headers\nto be added before the \"\\r\\n\\r\\n\" terminator at the body, we allow\nthem between \"\\0\".\n\nWhy '\\0'? The only real Git implementation that matters is C Git,\nand its written in C, and that's easy to work with in C.\n\nAs far as UTF-8 or other characters... that path is scanned to check\nfor nasty cases like '../../../../etc/passwd', but is otherwise\nhanded off to the system's stat() and chdir() functions as-is. So\nlike any other path in Git, it had damn well better match what the\nhost will recognize.\n\nIf the host is using SHIFT-JIS on its filesystem, then a client must\nrequest the path in SHIFT-JIS. And there is no way to specify that\nto the client in advance.\n\nIn practice, I think most people stick to an latin1 style character\nset here, maybe even the commonly acceptable printable characters\nfor US-ASCII, so it winds up being not that much of an issue.\n \n> Is the final terminating character required to be NUL (\"\\0\"), or can\n> it be for LF (\"\\n\"), i.e.\n> \n> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\n\n\nThe LF thing is like I said before, for a human, not the machine.\nHell, if the LF is present I think it would have to be *after* the\n'\\0' in the line, otherwise git daemon would assume that the host\nname includes an LF at the end of it.\n\nThe NUL at the end of the host name is not strictly required, but\nmust be present if the client were to ever pass additional options\nto the server.\n\nSee above about why... client reads line, sticks a NUL at the end,\nif the host header doesn't end in NUL on the wire, it does now\nin memory.\n \n> What options besides (required?) \"host=[:]\" are supported?\n\nCurrently only host is supported. And yea, it takes the : if\nthe client included the port number in the URL (git://foo:8813/path).\n\nActually, I just realized JGit isn't compliant here. It doesn't\nsend the : like C Git would.\n\n> Do I understand correctly that \"host=\" information is required\n> for core.gitProxy to work, isn't it?\n\nNo. Its for the git-daemon name based virtual hosting.\nSee --interpolated-path option to git daemon, with the %H/%CH\nformat characters.\n \n> >>>> * no-progress\n> >> \n> >> What that does mean?\n> > \n> > The client was started with \"git clone -q\" or something, and doesn't\n> > want that side brand 2. Basically the client just says \"I do not\n> > wish to receive stream 2 on sideband, so do not send it to me,\n> > and if you did, I will drop it on the floor anyway\".\n> \n> Does this mean that if server does not support \"no-progress\" capability\n> then client is required to drop diagnostic by itself?\n\nYes.\n\n> Can client request\n> to not use sideband (multiplexing) if it is asking for \"no-progress\";\n> or is multiplexing required for possible signaling of error condition \n> on channel 3?\n\nWe still want it for the error condition on channel 3. But if the\nclient didn't care about errors, and wanted no-progress, and the\nserver didn't support no-progress, then yes, the client could just\navoid asking for the side-band capability.\n\n> >> It is a bit pity that git protocol was not created with extendability\n> >> (like capabilities) in mind...\n> > \n> > Yes, no doubt. There are many things I would have done differently,\n> > given that I now have 20/20 hindsight vision into the past's future.\n> > \n> > :-)\n> > \n> > The protocol (mostly) works fine as-is. Its widely distributed in\n> > terms of clients using it on a daily basis. Its likely to continue\n> > to serve our needs well into the future. So, it is what it is.\n> \n> I do wonder if existing Internet Standard (in the meaning of RFC) \n> protocols also have such kludges and hacks...\n\nI'm sure they have some... oddities. But perhaps not as bad as git.\n\nWe have a history of not leaving ourselves room for future expansion,\nand then needing to find a backdoor in the canonical implementation\nparser in order to make it work.\n\nIn the protocol suite, its been the strlen() < pktlen trick which\nhas generally worked. Oh, and also sticking stuff after a fixed\nlength record, where we didn't care.\n\nOh, and send-pack/receive-pack protocol now has \".have\" refs, which\nwork for C Git because the send-pack client was always calling\ncheck_ref_format() on each thing sent by the server, and \".have\"\nisn't a valid ref name. Why the hell the send-pack client was doing\nthat, I have no idea. But, when the ref failed it was a silent\nfailure, so we were able to use \".have\" for some new capability.\n\nIt also broke JGit, which wasn't doing this seemingly pointless\ncheck_ref_format() and silently fail business. Oh, and IIRC,\nGitHub may have been burned around the same time somehow.\n\nIn packed-refs, Junio had a hard time adding the \"peeled-refs\"\nsupport, because the first version of the parser was so strict.\nBut again, somehow he managed to find a backdoor in the old parser,\nand that backdoor is why that file looks the way it does today.\n\nIn the loose object format, when we added new-style loose objects\nwe found a backdoor in the way libz deflate formats the first 2\nbytes of the file... and encoded something that shouldn't appear\nthere to signal it was a new \"pack style\" loose object.\n\nPack index v2 uses a hole where old clients would barf on the\n'\\377t0c' followed by the version '2' not being monotonically\nincreasing.\n\nI think there's something like that in DIRC too, but that change\n(to introduce the current DIRC format) may predate my involvement\nwith Git, so my memory isn't very good there.\n \n> P.S. By the way, is pkt-line format original invention, or was it \n> 'borrowed' from some other standard or protocol?\n\nNo clue. I find it f'king odd that the length is in hex. There\nisn't much value to the protocol being human readable. The PACK\npart of the stream sure as hell ain't. You aren't going to type\nout a sequence of \"have\" lines against the remote, like you could\nwith say an HTTP GET. *shrug*\n\n-- \nShawn.\n"},{"id":"115346","messageId":"20090603150746.GF3355@spearce.org","threadId":"19323","inReplyTo":"20090603144837.GE3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T15:07:46Z","receivedAt":"2009-06-03T15:07:46Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Shawn O. Pearce\" wrote:\n> Jakub Narebski wrote:\n> > What options besides (required?) \"host=[:]\" are supported?\n> \n> Currently only host is supported. And yea, it takes the : if\n> the client included the port number in the URL (git://foo:8813/path).\n\nOk, I'm wrong. It *doesn't* send the port. The reason is obtuse,\nbut git_tcp_connect() clobbers the port number out of the host\nname string, so that later when git_connect() sends this \"host=%s\",\nonly the host name is transmitted.\n \n> Actually, I just realized JGit isn't compliant here. It doesn't\n> send the : like C Git would.\n\nSo, actually JGit is compliant here.\n \n> > Do I understand correctly that \"host=\" information is required\n> > for core.gitProxy to work, isn't it?\n\nIf core.gitProxy or GIT_PROXY_COMMAND are set, you can lie to the\nremote git daemon about the host. E.g.:\n\n $ cat proxy.sh\n #!/bin/sh\n exec nc git.kernel.org 9418\n\n GIT_PROXY_COMMAND=proxy.sh git ls-remote git://github.com/foo.git\n\nDuring that kernel.org receives \"\\0host=github.com\\0\" host header,\nwhich is not the name you connected to it as. :-)\n\nIn practice I doubt anyone would do that, but, you can confuse\nyourself. I guess about equally as well as url.insteadof. :-)\n\n-- \nShawn.\n"},{"id":"115347","messageId":"200906031739.49682.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090603150746.GF3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T15:39:48Z","receivedAt":"2009-06-03T15:39:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> \"Shawn O. Pearce\" wrote:\n> > Jakub Narebski wrote:\n\n> > > What options besides (required?) \"host=[:]\" are supported?\n> > \n> > Currently only host is supported. And yea, it takes the : if\n> > the client included the port number in the URL (git://foo:8813/path).\n> \n> Ok, I'm wrong. It *doesn't* send the port. The reason is obtuse,\n> but git_tcp_connect() clobbers the port number out of the host\n\nWhat about git_proxy_connect()? Does it clobber port number either?\n\n> name string, so that later when git_connect() sends this \"host=%s\",\n> only the host name is transmitted.\n\nHmmm... so does that mean that in the following fragment of deamon.c\non branch is dead in practice?\n\n if (strncasecmp(\"host=\", extra_args, 5) == 0) {\n val = extra_args + 5;\n vallen = strlen(val) + 1;\n if (*val) {\n /* Split : at colon. */\n char *host = val;\n char *port = strrchr(host, ':');\n if (port) {\n *port = 0;\n port++;\n free(tcp_port);\n tcp_port = xstrdup(port);\n }\n free(hostname);\n hostname = xstrdup_tolower(host);\n }\n\n /* On to the next one */\n extra_args = val + vallen;\n }\n\n\n> > Actually, I just realized JGit isn't compliant here. It doesn't\n> > send the : like C Git would.\n> \n> So, actually JGit is compliant here.\n\nWell, we can take the stance that C Git isn't compliant either ;-)\n\n> \n> > > Do I understand correctly that \"host=\" information is required\n> > > for core.gitProxy to work, isn't it?\n> \n> If core.gitProxy or GIT_PROXY_COMMAND are set, you can lie to the\n> remote git daemon about the host. E.g.:\n> \n> $ cat proxy.sh\n> #!/bin/sh\n> exec nc git.kernel.org 9418\n> \n> GIT_PROXY_COMMAND=proxy.sh git ls-remote git://github.com/foo.git\n> \n> During that kernel.org receives \"\\0host=github.com\\0\" host header,\n> which is not the name you connected to it as. :-)\n> \n> In practice I doubt anyone would do that, but, you can confuse\n> yourself. I guess about equally as well as url.insteadof. :-)\n\n\nA question: do compliant implementation MUST not fail on receiving\narguments it doesn't understand, e.g.:\n\n 003bgit-upload-pack /project.git\\0host=myserver.com\\0user=me\\0\n\nor can it go hang the client, or silently fail?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115349","messageId":"20090603155023.GH3355@spearce.org","threadId":"19323","inReplyTo":"200906031739.49682.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T15:50:23Z","receivedAt":"2009-06-03T15:50:23Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> > \"Shawn O. Pearce\" wrote:\n> > \n> > Ok, I'm wrong. It *doesn't* send the port. The reason is obtuse,\n> > but git_tcp_connect() clobbers the port number out of the host\n> \n> What about git_proxy_connect()? Does it clobber port number either?\n\nDammit, not enough coffee.\n\nWe copy the string before the clobbering happens. It *DOES* send\nthe port. Either way, TCP or proxy.\n\nJGit isn't compliant. I'll send a patch soon.\n \n> > name string, so that later when git_connect() sends this \"host=%s\",\n> > only the host name is transmitted.\n> \n> Hmmm... so does that mean that in the following fragment of deamon.c\n> on branch is dead in practice?\n\nNo, its valid... I misread the client code.\n\n> A question: do compliant implementation MUST not fail on receiving\n> arguments it doesn't understand, e.g.:\n> \n> 003bgit-upload-pack /project.git\\0host=myserver.com\\0user=me\\0\n> \n> or can it go hang the client, or silently fail?\n\nMy understanding is that a compliant server MUST accept and ignore\nanything the client sends if it doesn't recognize it.\n\n-- \nShawn.\n"},{"id":"115353","messageId":"200906031851.12160.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090603144837.GE3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T16:51:11Z","receivedAt":"2009-06-03T16:51:11Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n\n[..]\n>>>> It is a bit pity that git protocol was not created with\n>>>> extendability (like capabilities) in mind...\n>>> \n>>> Yes, no doubt. There are many things I would have done differently,\n>>> given that I now have 20/20 hindsight vision into the past's future.\n>>> \n>>> :-)\n>>> \n>>> The protocol (mostly) works fine as-is. Its widely distributed in\n>>> terms of clients using it on a daily basis. Its likely to continue\n>>> to serve our needs well into the future. So, it is what it is.\n>> \n>> I do wonder if existing Internet Standard (in the meaning of RFC) \n>> protocols also have such kludges and hacks...\n\nI wonder if there are some BCP (Best Common Practices) RFC for designing\nprotocols (and BCP documents for designing file formats). And which one\nof RFC 2360, RFC 2424,... are applicable here.\n\n> \n> I'm sure they have some... oddities. But perhaps not as bad as git.\n> \n> We have a history of not leaving ourselves room for future expansion,\n> and then needing to find a backdoor in the canonical implementation\n> parser in order to make it work.\n> \n> In the protocol suite, its been the strlen() < pktlen trick which\n> has generally worked. Oh, and also sticking stuff after a fixed\n> length record, where we didn't care.\n\nMagic number (magic sequence) identifying protocol / format plus\nversion number. But it is good that we have capabilities now\n(which is better than version number in this case, IMHO).\n\n> \n> Oh, and send-pack/receive-pack protocol now has \".have\" refs, which\n> work for C Git because the send-pack client was always calling\n> check_ref_format() on each thing sent by the server, and \".have\"\n> isn't a valid ref name. Why the hell the send-pack client was doing\n> that, I have no idea. But, when the ref failed it was a silent\n> failure, so we were able to use \".have\" for some new capability.\n> \n> It also broke JGit, which wasn't doing this seemingly pointless\n> check_ref_format() and silently fail business. Oh, and IIRC,\n> GitHub may have been burned around the same time somehow.\n\nWhat are those \".have\" refs? They are not described in current version\nof \"Transfer Protocols\" (sub)section in \"The Community Book\". I remember\nthat they were discussed on git mailing list, but I don't remember what\nthey were about...\n\n> \n> In packed-refs, Junio had a hard time adding the \"peeled-refs\"\n> support, because the first version of the parser was so strict.\n> But again, somehow he managed to find a backdoor in the old parser,\n> and that backdoor is why that file looks the way it does today.\n\nI don't remember what that was about... Nevertheless now we have\nkind of 'capabilities' section in .git/packed-refs\n\n> \n> In the loose object format, when we added new-style loose objects\n> we found a backdoor in the way libz deflate formats the first 2\n> bytes of the file... and encoded something that shouldn't appear\n> there to signal it was a new \"pack style\" loose object.\n> \n> Pack index v2 uses a hole where old clients would barf on the\n> '\\377t0c' followed by the version '2' not being monotonically\n> increasing.\n\nInteresting... even more so that this problem of designing without\nextendability in mind (magic number + version) is so persistent :-(\n\n>> P.S. By the way, is pkt-line format original invention, or was it \n>> 'borrowed' from some other standard or protocol?\n> \n> No clue. I find it f'king odd that the length is in hex. There\n> isn't much value to the protocol being human readable. The PACK\n> part of the stream sure as hell ain't. You aren't going to type\n> out a sequence of \"have\" lines against the remote, like you could\n> with say an HTTP GET. *shrug*\n\nWell... in theory you could... ;-)\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115354","messageId":"20090603165613.GJ3355@spearce.org","threadId":"19323","inReplyTo":"200906031851.12160.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T16:56:13Z","receivedAt":"2009-06-03T16:56:13Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> > Oh, and send-pack/receive-pack protocol now has \".have\" refs [...]\n> \n> What are those \".have\" refs? They are not described in current version\n> of \"Transfer Protocols\" (sub)section in \"The Community Book\". I remember\n> that they were discussed on git mailing list, but I don't remember what\n> they were about...\n\nIf the remote receiving repository has alternates, the \".have\" refs are\nthe refs of the alternate repositories. This signals to the client that\nthe server has these objects reachable, but the client isn't permitted\nto send commands to alter these refs.\n\nIts good for a site like GitHub or repo.or.cz where cheap forks are\nimplemented by creating a repository that points to a common shared\nbase via alternates. The \".have\" refs say that the server already\nhas everything in that common shared base, so the client doesn't\nhave to re-upload the entire project if the fork started out empty,\nor had all refs deleted from it.\n \n> > In packed-refs, Junio had a hard time adding the \"peeled-refs\"\n> > support, because the first version of the parser was so strict.\n> > But again, somehow he managed to find a backdoor in the old parser,\n> > and that backdoor is why that file looks the way it does today.\n> \n> I don't remember what that was about... Nevertheless now we have\n> kind of 'capabilities' section in .git/packed-refs\n\nSort of. In a file format its worse than a network protocol,\nbecause the file can't alter its contents based on what the\nreader can understand.\n \n> Interesting... even more so that this problem of designing without\n> extendability in mind (magic number + version) is so persistent :-(\n\nI know. I think we maybe have learned the lesson. I don't know.\n \n-- \nShawn.\n"},{"id":"115355","messageId":"7vtz2x9r9i.fsf@alter.siamese.dyndns.org","threadId":"19323","inReplyTo":"20090603144837.GE3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-06-03T17:11:21Z","receivedAt":"2009-06-03T17:11:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> I think there's something like that in DIRC too, but that change\n> (to introduce the current DIRC format) may predate my involvement\n> with Git, so my memory isn't very good there.\n\nThe index_extension part where the cache-tree is stored is another\nexample. The length of the index is known from the mmap, the file\nchecksum is defined to appear at the end of the file, and the number of\nentries are recorded in the file header, so there was a hole after that\nmany index entries I could add new section.\n\n>> P.S. By the way, is pkt-line format original invention, or was it \n>> 'borrowed' from some other standard or protocol?\n>\n> No clue. I find it f'king odd that the length is in hex. There\n> isn't much value to the protocol being human readable. The PACK\n> part of the stream sure as hell ain't. You aren't going to type\n> out a sequence of \"have\" lines against the remote, like you could\n> with say an HTTP GET. *shrug*\n\nThe text-ness made it easier to debug while I was developing the sideband\nsupport. I literally typed the pkt-line from the terminal ;-).\n"},{"id":"115360","messageId":"200906032105.18459.j6t@kdbg.org","threadId":"19323","inReplyTo":"20090603144837.GE3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2009-06-03T19:05:18Z","receivedAt":"2009-06-03T19:05:18Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Mittwoch, 3. Juni 2009, Shawn O. Pearce wrote:\n> We have a history of not leaving ourselves room for future expansion,\n> and then needing to find a backdoor in the canonical implementation\n> parser in order to make it work.\n>\n> In the protocol suite, its been the strlen() < pktlen trick which\n> has generally worked. Oh, and also sticking stuff after a fixed\n> length record, where we didn't care.\n\nThis reminds me of one thing: upload-pack (of C git) sends a complete pack if \nand only if there were no errors, so that fetch-pack sees an error if \nupload-pack dies or if there is no side-band where upload-pack could signal \nan error (at least I think that are the reasons). There is a comment in \nupload-pack that explains a bit of it:\n\n/* Data ready; we keep the last byte to ourselves\n * in case we detect broken rev-list, so that we\n * can leave the stream corrupted. This is\n * unfortunate -- unpack-objects would happily\n * accept a valid packdata with trailing garbage,\n * so appending garbage after we pass all the\n * pack data is not good enough to signal\n * breakage to downstream.\n */\n\n-- Hannes\n"},{"id":"115368","messageId":"200906032220.00238.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090603165613.GJ3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T20:19:57Z","receivedAt":"2009-06-03T20:19:57Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n>> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n\n>>> Oh, and send-pack/receive-pack protocol now has \".have\" refs [...]\n>> \n>> What are those \".have\" refs? They are not described in current version\n>> of \"Transfer Protocols\" (sub)section in \"The Community Book\". I remember\n>> that they were discussed on git mailing list, but I don't remember what\n>> they were about...\n> \n> If the remote receiving repository has alternates, the \".have\" refs are\n> the refs of the alternate repositories. This signals to the client that\n> the server has these objects reachable, but the client isn't permitted\n> to send commands to alter these refs.\n> \n> Its good for a site like GitHub or repo.or.cz where cheap forks are\n> implemented by creating a repository that points to a common shared\n> base via alternates. The \".have\" refs say that the server already\n> has everything in that common shared base, so the client doesn't\n> have to re-upload the entire project if the fork started out empty,\n> or had all refs deleted from it.\n\nSo the output (for fetch or clone) would look like this for repository\nwith alternates (shared object database):\n\n 00887b68fcd777f94534f0b794c5dc2e109c49938395 HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag\\n\n 0048a6afbb5c9618395ed28a299f0913e9e6df2058ef refs/heads/adaptive-filter\\n\n 004bbc60ab88b6899573fb545c6b4961f8ff3ce20695 refs/heads/filtered-to-window\\n\n 003f7b68fcd777f94534f0b794c5dc2e109c49938395 refs/heads/master\\n\n 0044226d09c3b5e16b5c1bd377aae9459cae3f778847 refs/heads/save-config\\n\n 0050dab192738152e1fa7233e06d941f9ada865c6e65 refs/tags/jnareb@gmail.com-gpg-pub\\n\n 00535812582f41a234828c8a2ec38047462979dc5dd8 refs/tags/jnareb@gmail.com-gpg-pub^{}\\n\n 003c50808bc27817eac090683e44fce4368fff39f9b2 refs/tags/v1.2\\n\n 0033b11cf09043f18b368ec0d988f064ea21247c843d .keep\\n\n\nDoes it matter for fetch, or is it important only for pushing?\n\n\nBTW. do \"include-tag\" capability MUST NOT (REQUIRED) be send if there\nare not tags (tag objects?), or just SHOULD NOT (RECOMMENDED), or even\nMAY NOT (OPTIONAL). GitHub server doesn't send it if there are no \ntags...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115371","messageId":"20090603202429.GO3355@spearce.org","threadId":"19323","inReplyTo":"200906032220.00238.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T20:24:29Z","receivedAt":"2009-06-03T20:24:29Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \n> >>> Oh, and send-pack/receive-pack protocol now has \".have\" refs [...]\n> \n> So the output (for fetch or clone) would look like this for repository\n> with alternates (shared object database):\n\nNo. fetch/clone (aka fetch-pack/upload-pack protocl) does not use\nthe .have feature.\n \n> Does it matter for fetch, or is it important only for pushing?\n\nBecause yea, it only matters for pushing. Actually, in the case of\nfetch, we shouldn't advertise what our alternate has, the client\nshould just fetch from the alternate.\n\nIn push it matters because the client wants to know what the remote\nhas, so it can trim the pack down to only the new objects, to reduce\ntransfer time.\n \n> BTW. do \"include-tag\" capability MUST NOT (REQUIRED) be send if there\n> are not tags (tag objects?), or just SHOULD NOT (RECOMMENDED), or even\n> MAY NOT (OPTIONAL). GitHub server doesn't send it if there are no \n> tags...\n\nClients MAY always send include-tag, hardcoding it into a request.\nThe decision for a client to request include-tag only has to do\nwith the client's desires for tag data, whether or not a server\nhad advertised objects in the refs/tags/* namespace.\n\nClients SHOULD NOT send include-tag if remote.name.tagopt was set\nto --no-tags, as the client doesn't want tag data.\n\nServers MUST accept include-tag without error or warning, even if the\nserver does not understand or support the option.\n\nServers SHOULD pack the tags if their referrant is packed and the\nclient has requested include-tag.\n\nClients MUST be prepared for the case where a server has ignored\ninclude-tag and has not actually sent tags in the pack. In such\ncases the client SHOULD issue a subsequent fetch to acquire the\ntags that include-tag would have otherwise given the client.\n\n-- \nShawn.\n"},{"id":"115374","messageId":"alpine.LSU.2.00.0906032147330.31588@hermes-2.csi.cam.ac.uk","threadId":"19323","inReplyTo":"200906030250.01413.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-06-03T20:56:07Z","receivedAt":"2009-06-03T20:56:07Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Wed, 3 Jun 2009, Jakub Narebski wrote:\n>\n> HEXDIGIT = 0-9 / a-f\n>\n> Well, it should probably be spelled in full. Probably, because I have\n> no experience with using ABNF... and didn't do my research :-)\n\nThe ABNF core rules include a definition for HEXDIG. See appendix B of\nRFC 5234.\n\n> (should HEXDIGIT use lowercase a-f, or can it use uppercase A-F?)\n\nDouble-quoted strings in ABNF are case-insensitive ASCII, so the HEXDIG\nrule accepts both. You need to use %x61 if you want a but not A.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"115376","messageId":"200906032320.23559.jnareb@gmail.com","threadId":"19323","inReplyTo":"alpine.LSU.2.00.0906032147330.31588@hermes-2.csi.cam.ac.uk","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T21:20:19Z","receivedAt":"2009-06-03T21:20:19Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 3 Jan 2009, Tony Finch wrote:\n> On Wed, 3 Jun 2009, Jakub Narebski wrote:\n> >\n> > HEXDIGIT = 0-9 / a-f\n> >\n> > Well, it should probably be spelled in full. Probably, because I have\n> > no experience with using ABNF... and didn't do my research :-)\n> \n> The ABNF core rules include a definition for HEXDIG. See appendix B of\n> RFC 5234.\n\nI have found it _after_ sending this post in Wikipedia article (which\nis shorter than RFC 5234), but thanks anyway.\n\n> > (should HEXDIGIT use lowercase a-f, or can it use uppercase A-F?)\n> \n> Double-quoted strings in ABNF are case-insensitive ASCII, so the HEXDIG\n> rule accepts both. You need to use %x61 if you want a but not A.\n\n HEXDIG_LC = DIGIT / %x61-%x66 ; 0-9a-f, case sensitive\n\nActually git accepts both lowercase and uppercase in HEXDIG (at least\nfor pkt-length), but it prefers lowercase.\n-- \nJakub Narebski\nPoland\n"},{"id":"115380","messageId":"alpine.LSU.2.00.0906032211350.31588@hermes-2.csi.cam.ac.uk","threadId":"19323","inReplyTo":"200906031851.12160.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-06-03T21:38:31Z","receivedAt":"2009-06-03T21:38:31Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Wed, 3 Jun 2009, Jakub Narebski wrote:\n> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> > Jakub Narebski wrote:\n> >>\n> >> I do wonder if existing Internet Standard (in the meaning of RFC)\n> >> protocols also have such kludges and hacks...\n> >\n> > I'm sure they have some... oddities. But perhaps not as bad as git.\n\nOne small example is the way DNS was extended to support extra flags and\nresponse codes (and other features). EDNS is signalled using an OPT\npseudo-RR, which is basically the same technique as git's .have refs.\n\nThere are a couple of examples in RFC 822 / MIME headers: RFC 2047 (for\nencoding character set information in subject and address headers) and RFC\n2231 (the same job but for attachment filenames etc.). In practice common\nsoftware uses 2047 syntax for both purposes :-/\n\nMostly Internet protocols have grown generic extension frameworks fairly\nearly in their lives, so syntactic hacks are rare.\n\n> I wonder if there are some BCP (Best Common Practices) RFC for designing\n> protocols (and BCP documents for designing file formats). And which one\n> of RFC 2360, RFC 2424,... are applicable here.\n\nAs far as I know protocol design is pretty much folklore.\n\n> Magic number (magic sequence) identifying protocol / format plus\n> version number. But it is good that we have capabilities now\n> (which is better than version number in this case, IMHO).\n\nYes, capabilities are a good design pattern.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"115381","messageId":"alpine.LSU.2.00.0906032240300.31588@hermes-2.csi.cam.ac.uk","threadId":"19323","inReplyTo":"200906032320.23559.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-06-03T21:53:20Z","receivedAt":"2009-06-03T21:53:20Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Wed, 3 Jun 2009, Jakub Narebski wrote:\n>\n> Actually git accepts both lowercase and uppercase in HEXDIG (at least\n> for pkt-length), but it prefers lowercase.\n\nYou should ensure that all hex digit strings follow the same rule.\nAre SHA-1 object names case insensitive too?\n\nCase insensitivity has a history of being awkward. SMTP has always had\ncase-insensitive commands, though the RFCs have always written them in\nupper case and implementations have pretty much all emitted them in upper\ncase. See http://tools.ietf.org/html/rfc5321#section-2.4 especially the\ncaveat about broken case-sensitive implementations.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"115384","messageId":"200906040004.18594.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090603202429.GO3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T22:04:09Z","receivedAt":"2009-06-03T22:04:09Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n\n> > BTW. do \"include-tag\" capability MUST NOT (REQUIRED) be send if there\n> > are not tags (tag objects?), or just SHOULD NOT (RECOMMENDED), or even\n> > MAY NOT (OPTIONAL). GitHub server doesn't send it if there are no \n> > tags...\n> \n> Clients MAY always send include-tag, hardcoding it into a request.\n> The decision for a client to request include-tag only has to do\n> with the client's desires for tag data, whether or not a server\n> had advertised objects in the refs/tags/* namespace.\n> \n> Clients SHOULD NOT send include-tag if remote.name.tagopt was set\n> to --no-tags, as the client doesn't want tag data.\n> \n> Servers MUST accept include-tag without error or warning, even if the\n> server does not understand or support the option.\n> \n> Servers SHOULD pack the tags if their referrant is packed and the\n> client has requested include-tag.\n> \n> Clients MUST be prepared for the case where a server has ignored\n> include-tag and has not actually sent tags in the pack. In such\n> cases the client SHOULD issue a subsequent fetch to acquire the\n> tags that include-tag would have otherwise given the client.\n\nSo do I understand correctly that capabilities are governed by the\nfollowing generic rules:\n\n1. Server sends space separated list of capabilities it support. It\n MUST NOT send capabilities it *does not* support. It MAY NOT send\n \"include-tag\" if there are no tag objects (or is it SHOULD NOT?).\n2. Client sends space separated list of capabilities it wants. It SHOULD\n (or perhaps it is MAY?) send subset of server capabilities, i.e do\n not send capabilities served does not advertise.\n3. Server MUST ignore capabilities it does not understand. Server MUST\n NOT ignore capabilities (or SHOULD NOT only?) that client requested\n and server advertised.\n\nI know that client MUST send only maximum of one of \"side-band\" and \n\"side-band-64k\", but how should server reacts if client sends both?\nShould it use \"side-band-64k\"?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115386","messageId":"20090603220456.GQ3355@spearce.org","threadId":"19323","inReplyTo":"200906040004.18594.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-03T22:04:56Z","receivedAt":"2009-06-03T22:04:56Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> So do I understand correctly that capabilities are governed by the\n> following generic rules:\n> \n> 1. Server sends space separated list of capabilities it support. It\n> MUST NOT send capabilities it *does not* support. It MAY NOT send\n> \"include-tag\" if there are no tag objects (or is it SHOULD NOT?).\n\nThe server SHOULD send include-tag, if it supports it, irregardless\nof whether or not there are tags available. Its just easier to\ncode the server to send the R@!* line up front based on the software\nversion, and not the repository content.\n\n> 2. Client sends space separated list of capabilities it wants. It SHOULD\n> (or perhaps it is MAY?) send subset of server capabilities, i.e do\n> not send capabilities served does not advertise.\n\nIt SHOULD send a subset of server capabilities.\n\n> 3. Server MUST ignore capabilities it does not understand.\n\nTrue.\n\n> Server MUST NOT ignore capabilities (or SHOULD NOT only?) that\n> client requested and server advertised.\n\nTrue, MUST NOT. Otherwise you will have protocol errors.\n\nHowever, include-tag can be SHOULD NOT... since the client must be\nable to recover from it anyway.\n\n> I know that client MUST send only maximum of one of \"side-band\" and \n> \"side-band-64k\", but how should server reacts if client sends both?\n> Should it use \"side-band-64k\"?\n\nMUST favor side-band-64k if client requests both.\n\n-- \nShawn.\n"},{"id":"115387","messageId":"7vzlcp55g3.fsf@alter.siamese.dyndns.org","threadId":"19323","inReplyTo":"200906040004.18594.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-06-03T22:16:12Z","receivedAt":"2009-06-03T22:16:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> 1. Server sends space separated list of capabilities it support. It\n> MUST NOT send capabilities it *does not* support. It MAY NOT send\n> \"include-tag\" if there are no tag objects (or is it SHOULD NOT?).\n\nI doubt RFC 2119 lingo would include MAY NOT, as it is ambiguous\nespecially to non-native speakers (like me). You meant to say \"MAY omit\nsending\", perhaps, but in general capabilies are what you _can_ do at the\nprotocol level, and in my opinion, you shouldn't have to check if a\nparticular repository you (as a program with given set of features\nimplemented) happen to be looking at has tags in order to decide what\ncapabilities to advertise.\n\n> 2. Client sends space separated list of capabilities it wants. It SHOULD\n> (or perhaps it is MAY?) send subset of server capabilities, i.e do\n> not send capabilities served does not advertise.\n\nI'd say \"the client SHOULD NOT ask for capabilities the server did not say\nit supports\".\n\n> 3. Server MUST ignore capabilities it does not understand. Server MUST\n> NOT ignore capabilities (or SHOULD NOT only?) that client requested\n> and server advertised.\n\nI know unrecognized capability requests are silently ignored, but I\nconsider that as a sloppy/practical programming, and not a specification.\n"},{"id":"115390","messageId":"200906040046.22383.jnareb@gmail.com","threadId":"19323","inReplyTo":"7vzlcp55g3.fsf@alter.siamese.dyndns.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-03T22:46:21Z","receivedAt":"2009-06-03T22:46:21Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n> Jakub Narebski writes:\n> \n> > 1. Server sends space separated list of capabilities it support. It\n> > MUST NOT send capabilities it *does not* support. It MAY NOT send\n> > \"include-tag\" if there are no tag objects (or is it SHOULD NOT?).\n> \n> I doubt RFC 2119 lingo would include MAY NOT, as it is ambiguous\n> especially to non-native speakers (like me). \n\nYou are right, RFC 2119 does not include MAY NOT.\n\n> You meant to say \"MAY omit \n> sending\", perhaps, but in general capabilies are what you _can_ do at the\n> protocol level, and in my opinion, you shouldn't have to check if a\n> particular repository you (as a program with given set of features\n> implemented) happen to be looking at has tags in order to decide what\n> capabilities to advertise.\n\nI wonder why in http://book.git-scm.com/7_transfer_protocols.html \n(\"Git Community Book\", chapter \"7. Internals and Plumbing\", section\n \"Transfer Protocols\", subsection \"Fetching Data with Upload Pack\")\n\"include-tag\" is not included ;) in advertised server capabilities.\nBecause github's git-daemon advertises it even if there are no\ntags present\n\n $ echo -e -n \"0039git-upload-pack /schacon/gitbook.git\\0host=github.com\\0\" | \n nc -v github.com 9418\n Connection to github.com 9418 port [tcp/*] succeeded!\n 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag\n 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n 003f7217a7c7e582c46cec22a130adf4b9d7d950fba0 refs/heads/master\n 003edc9d991bc43cb04e692efc793f885eb4ff7fda98 refs/heads/pt_BR\n\n> > 2. Client sends space separated list of capabilities it wants. It SHOULD\n> > (or perhaps it is MAY?) send subset of server capabilities, i.e do\n> > not send capabilities served does not advertise.\n> \n> I'd say \"the client SHOULD NOT ask for capabilities the server did not say\n> it supports\".\n\nI agree that it is better formulation (phrasing).\n\n> \n> > 3. Server MUST ignore capabilities it does not understand. Server MUST\n> > NOT ignore capabilities (or SHOULD NOT only?) that client requested\n> > and server advertised.\n> \n> I know unrecognized capability requests are silently ignored, but I\n> consider that as a sloppy/practical programming, and not a specification.\n\nWell, the whole 'be strict in what you send, and accepting in what you\naccept' leads unfortunately to accepting sloppy programming and coding.\n\nNevertheless it is I guess better to silently ignore unknown\ncapabilities requested by client than fail, isn't it?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115414","messageId":"4A277507.2060409@op5.se","threadId":"19323","inReplyTo":"20090603202429.GO3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-06-04T07:17:27Z","receivedAt":"2009-06-04T07:17:27Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> \n> Servers MUST accept include-tag without error or warning, even if the\n> server does not understand or support the option.\n> \n> Servers SHOULD pack the tags if their referrant is packed and the\n> client has requested include-tag.\n> \n> Clients MUST be prepared for the case where a server has ignored\n> include-tag and has not actually sent tags in the pack. In such\n> cases the client SHOULD issue a subsequent fetch to acquire the\n> tags that include-tag would have otherwise given the client.\n> \n\nHow is \"no tags present\" signalled? Without such a signal, the client\nmust always issue a subsequent request every time there are no tags\nembedded in the received pack, as it can't know if the server ignored\nthe option silently or if there just aren't any new tags.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"115417","messageId":"7v63fczchb.fsf@alter.siamese.dyndns.org","threadId":"19323","inReplyTo":"4A277507.2060409@op5.se","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-06-04T07:26:08Z","receivedAt":"2009-06-04T07:26:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Ericsson writes:\n\n> How is \"no tags present\" signalled? Without such a signal, the client\n> must always issue a subsequent request every time there are no tags\n> embedded in the received pack, as it can't know if the server ignored\n> the option silently or if there just aren't any new tags.\n\nThe fetcher first learns the set of tags and what objects they point at.\nThat's in the first part of the upload-pack protocol.\n\nOf course, if there is no tag, you won't see them advertised, so you can\nknow.\n\nAfter finishing the main part of the object transfer, if some of the\nobjects that are pointed at by tags are present (and reachable from refs),\nbut the tags that point at them do not exist yet, that is a sign that the\nuploader didn't give you these tag objects. Then you can turn around to\nrequest those tags by initiating another exchange, and ask for them with\n\"want\" lines.\n"},{"id":"115419","messageId":"200906041045.11328.jnareb@gmail.com","threadId":"19323","inReplyTo":"alpine.LSU.2.00.0906032240300.31588@hermes-2.csi.cam.ac.uk","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-04T08:45:07Z","receivedAt":"2009-06-04T08:45:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia środa 3. czerwca 2009 23:53, Tony Finch napisał:\n> On Wed, 3 Jun 2009, Jakub Narebski wrote:\n> >\n> > Actually git accepts both lowercase and uppercase in HEXDIG (at least\n> > for pkt-length), but it prefers lowercase.\n> \n> You should ensure that all hex digit strings follow the same rule.\n> Are SHA-1 object names case insensitive too?\n> \n> Case insensitivity has a history of being awkward. SMTP has always had\n> case-insensitive commands, though the RFCs have always written them in\n> upper case and implementations have pretty much all emitted them in upper\n> case. See http://tools.ietf.org/html/rfc5321#section-2.4 especially the\n> caveat about broken case-sensitive implementations.\n\nThere should be no problem with pkt-length being case insensitive, as\nstandard conversion routines (strtol, sprintf) accept 0-9a-fA-F for \nbase 16 / hexadecimal conversion. The requirement here is that client\nand server SHOULD use lowercase, but MUST accept mixed case (do case\ninsensitive parsing of hex4).\n\nI think SHA-1 is lowercased, so mixed case should work there. Well, at\nleast \"git show 6096D7\" (note the uppercase 'D' in shortened SHA-1 name)\nworks as expected.\n\n\nBut I do not know what are, or what should be protocol requirements.\nShould SHA-1 use lowercase, or be case insensitive? Should commands such\nas \"have\", \"want\", \"done\" use lower case or be case insensitive? Should\nstatus indicators \"ACK\" and \"NAK\" be upper case, or should be case\ninsensitive? Should capabilities be case sensitive, and should they be\ncompared case sensitive or not?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115423","messageId":"alpine.LSU.2.00.0906041157221.9226@hermes-2.csi.cam.ac.uk","threadId":"19323","inReplyTo":"200906041045.11328.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-06-04T11:41:14Z","receivedAt":"2009-06-04T11:41:14Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Thu, 4 Jun 2009, Jakub Narebski wrote:\n>\n> But I do not know what are, or what should be protocol requirements.\n> Should SHA-1 use lowercase, or be case insensitive? Should commands such\n> as \"have\", \"want\", \"done\" use lower case or be case insensitive? Should\n> status indicators \"ACK\" and \"NAK\" be upper case, or should be case\n> insensitive? Should capabilities be case sensitive, and should they be\n> compared case sensitive or not?\n\nI think the current (rough) consensus is that case insensitivity causes\npain unless its scope is carefully controlled. I18n causes a lot of the\ndifficulties.\n\nOne way in which you can control the scope is by limiting\ncase-insensitivity to protocol elements that must be ASCII (commands,\nreplies, SHA-1 hashes). But I'm not sure there's any benefit to making\nthe protocol case insensitive, especially when it isn't possible to\ntype it manually.\n\nI've already given one example of interoperability problems in SMTP\narising from case insensitivity. In the opposite direction, Unix and\nXML are good examples of case sensitivity working well in practice.\n\nI have to say I spend all my time working with old-school case insensitive\nprotocols, and they have clearly been extremely successful, so it's\ntempting to copy them. But I think that will lead to ugliness - have a\nlook through the HTTP spec for its mish-mash of sensitive and insensitive\nprotocol elements.\n\nIn the specific instance of the pkt-length, if all current implementations\nwrite the length in lower case, you can say in the spec it MUST be lower\ncase. If you do that then the same requirement can apply to both the\nclient and the server which makes the spec shorter and simpler. Postel's\nrobustness principle suggests that it doesn't really matter if the parser\ntreats it case-insensitively.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"115437","messageId":"20090604184109.GR3355@spearce.org","threadId":"19323","inReplyTo":"200906041045.11328.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-04T18:41:09Z","receivedAt":"2009-06-04T18:41:09Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> Dnia ?roda 3. czerwca 2009 23:53, Tony Finch napisa?:\n> > On Wed, 3 Jun 2009, Jakub Narebski wrote:\n> > >\n> > > Actually git accepts both lowercase and uppercase in HEXDIG (at least\n> > > for pkt-length), but it prefers lowercase.\n> > \n> > You should ensure that all hex digit strings follow the same rule.\n> > Are SHA-1 object names case insensitive too?\n> > \n> > Case insensitivity has a history of being awkward. SMTP has always had\n> > case-insensitive commands, though the RFCs have always written them in\n> > upper case and implementations have pretty much all emitted them in upper\n> > case. See http://tools.ietf.org/html/rfc5321#section-2.4 especially the\n> > caveat about broken case-sensitive implementations.\n> \n> There should be no problem with pkt-length being case insensitive, as\n> standard conversion routines (strtol, sprintf) accept 0-9a-fA-F for \n> base 16 / hexadecimal conversion. The requirement here is that client\n> and server SHOULD use lowercase, but MUST accept mixed case (do case\n> insensitive parsing of hex4).\n\nACK. This is what C Git does today. JGit sends lower case, but\nis wrong by only accepting lowercase. I will patch it today to\naccept mixed case.\n \n> I think SHA-1 is lowercased, so mixed case should work there. Well, at\n> least \"git show 6096D7\" (note the uppercase 'D' in shortened SHA-1 name)\n> works as expected.\n\nACK. Mixed case SHA-1 MUST be accepted, but lower case SHOULD\nbe output.\n \n> But I do not know what are, or what should be protocol requirements.\n> Should SHA-1 use lowercase, or be case insensitive?\n\nSHA-1 SHOULD be lowercase (a-f), MUST accept a-f or A-F.\n\n> Should commands such as \"have\", \"want\", \"done\" use lower case or\n> be case insensitive?\n\nThese MUST be lowercase.\n\n> Should status indicators \"ACK\" and \"NAK\" be upper case,\n\nThese MUST be uppercase. Though \"ACK %s continue\" MUST be mixed\ncase, as I just wrote it.\n\n> Should capabilities be case sensitive, and should they be\n> compared case sensitive or not?\n\nNo, they are case sensitive. \n\nWhy? All of the above is the current C code implementation.\nWe have to follow what the code does today, and it does case\nsensitive compares almost everywhere... except in the SHA-1 parsing.\n\n-- \nShawn.\n"},{"id":"115443","messageId":"200906042255.43952.jnareb@gmail.com","threadId":"19323","inReplyTo":"200906022339.08639.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-04T20:55:43Z","receivedAt":"2009-06-04T20:55:43Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is combined response to various messages in this thread, following\nmy discoveries done using simple Perl script (using IO::Socket) which\nassumes role of a git client, tested against github.com (IIRC it uses\nRuby implementation) and git.kernel.org (C Git), and \"nc -l 9418\".\n\nBy the way, is there some publicly accessible JGit (Java) and Dulwich\n(Python) git-daemon one can test against?\n\n sp = Shawn O. Pearce\n jn = Jakub Narebski\n gb = Git Community Book (http://book.git-scm.com)\n\n\njn>> I meant that in the request line for fetching via git:// protocol\njn>>\njn>>       0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\njn>>\njn>> you separate path to repository from extra options using \"\\0\" / NUL\njn>> as a separator. Well, this is only sane separator, as it is path\njn>> terminator, the only character which cannot appear in pathname\njn>> (although I do wonder whether project names with e.g. control\njn>> characters or UTF-8 characters would work correctly).\nsp>\nsp> No, that isn't the reason '\\0' is used here.  But yea, that is true.\nsp>\nsp> The reason \\0 is used is, git-daemon reads the 4 byte length, decodes\nsp> that, then reads that many bytes.  Finally it writes a '\\0' at the\nsp> end of what it read, so that the entire \"line\" is NUL terminated.\nsp> Then it reads the \"command path\" part from the resulting C string.\nsp>\nsp> The host=myserver.com part came later, after many daemons were\nsp> already running all over the world.  By hiding it behind the '\\0'\nsp> an old daemon would never see it (but strlen() returned a value that\nsp> was less than the length read, but the old daemons didn't care).\nsp> Newer daemons look for where strlen() < length, and assume that\nsp> the host header follows.\nsp>\nsp> The host header ends with '\\0' in case additional headers would\nsp> also appear here in the future.  IOW, like HTTP allows new headers\nsp> to be added before the \"\\r\\n\\r\\n\" terminator at the body, we allow\nsp> them between \"\\0\".\n[...]\n\nsp> The NUL at the end of the host name is not strictly required, but\nsp> must be present if the client were to ever pass additional options\nsp> to the server.\n\nActually both git.kernel.org and github.com failed (deadlocked / hung)\nwhen I tried to add extra key=value parameter at the end of request:\n\n 003bgit-upload-pack /project.git\\0host=myserver.com\\0user=me\\0\n\nHmmmm...\n\n\njn>> Hmmm... the communication between server and client is not entirely\njn>> clean. Do I understand correctly that this NAK is response to\njn>> clients flush after all those \"want\" lines?\nsp>\nsp> Yes.\nsp>\njn>> And that \"0009done\" from client\njn>> tells server that it should send everything it has?\nsp>\nsp> Yes.  It means the client will not issue any more \"have\" lines,\nsp> as it has nothing further in its history, so the server just has\nsp> to give up and start generating a pack based on what it knows.\n\nHere we were talking about the following part of exchange: \n(I have added \"C:\" prefix to signal that this is what client, \ngit-clone here, sends; I have added also explicit \"\\n\" to mark LF\ncharacters terminating lines, and put each pkt-line on separate line)\n\ngb> C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\\n\ngb> C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\n\ngb> C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\n\ngb> C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\n\ngb> C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\n\ngb> C: 0000\ngb> C: 0009done\\n\n\nand where server response is (again the quote from \"Git Community Book\"\nwas modified, removing here doublequotes and doubling of backslashes):\n\ngb> S: 0008NAK\\n\ngb> S: 0023\\002Counting objects: 2797, done.\\n\ngb> [...]\ngb> S: 2004\\001PACK\\000\\000\\000\\002 [...]\n\nI have thought that after sending \"0000\" flush line client can wait for\nNAK or ACK server response... but it is not the case. When I tried to\nread from server after \"0000\" flush and before \"0009done\\n\", my client\n(or netcat instance) deadlocked (hung) waiting for server response.\nI either did a mistake in my fake client, or I don't understand git pack\nprotocol correctly. Should client wait for NAK or ACK from server _only_\nafter sending maximum number of want/have lines (256 if I remember \ncorrectly?)?\n\nWhen I removed sending \"0000\" flush line my fake client again hung \n(deadlocked?) waiting for server.\n\n\njn>> P.S. By the way, is pkt-line format original invention, or was it \njn>> 'borrowed' from some other standard or protocol?\nsp>\nsp> No clue. I find it f'king odd that the length is in hex. There\nsp> isn't much value to the protocol being human readable. The PACK\nsp> part of the stream sure as hell ain't. You aren't going to type\nsp> out a sequence of \"have\" lines against the remote, like you could\nsp> with say an HTTP GET. *shrug*\n\n\"git gui blame pkt-line.c\" shows that pkt-line format is Linus invention.\n\nIt looks quite a bit like 'chunked' transfer encoding[1] in HTTP; there\neach non-empty chunk starts with the number of octets of the data it\nembeds (size written in hexadecimal) followed by a CRLF (carriage return\nand linefeed), and the data itself. The chunk is then closed with a CRLF.\nIn some implementations, white space chars (0x20) are padded between\nchunk-size and the CRLF. In pkt-line format number of octet has fixed\nwidth (4 hexadecimal digits, 0-padded), and we do not use CRLF as \nterminator of chunk/packet length and of chunk/packet itself.\n\nIn HTTP 'chunked' transfer encoding the last chunk is a single line,\nsimply made of the chunk-size (0). In pkt-line format we use special\nsize of \"0000\" for a flush packet.\n\n[1] http://en.wikipedia.org/wiki/Chunked_transfer_encoding\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115455","messageId":"20090604215748.GS3355@spearce.org","threadId":"19323","inReplyTo":"200906042255.43952.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-04T21:57:48Z","receivedAt":"2009-06-04T21:57:48Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \n> By the way, is there some publicly accessible JGit (Java) and Dulwich\n> (Python) git-daemon one can test against?\n\nThere is a public JGit SSH based daemon running at\nreview.source.android.com, on port 29418. But no\npublic git-daemon that I know of.\n\nYou can easily set up a JGit daemon yourself, assuming you have a\n'javac' available:\n\n git clone git://repo.or.cz/egit.git jgit\n cd jgit\n ./make_jgit.sh\n ./jgit daemon --export-all . &\n git ls-remote git://localhost/.git\n \n> Actually both git.kernel.org and github.com failed (deadlocked / hung)\n> when I tried to add extra key=value parameter at the end of request:\n> \n> 003bgit-upload-pack /project.git\\0host=myserver.com\\0user=me\\0\n\nJGit does it fine. I retested locally with this:\n\n perl -e '$s=\"git-upload-pack $ARGV[0]\\0hosterver.com\\0user=me\\0\";\n printf \"%4.4x%s\",4+length $s,$s\n ' /.git | nc localhost 9418\n\nBut yea, repo.or.cz hung. I see the bug in git daemon, I'll post\na patch in a second. Don't do that test anymore, anywhere.\n\n-- \nShawn.\n"},{"id":"115462","messageId":"20090605004540.GU3355@spearce.org","threadId":"19323","inReplyTo":"200906042255.43952.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-05T00:45:40Z","receivedAt":"2009-06-05T00:45:40Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \n> Here we were talking about the following part of exchange: \n> (I have added \"C:\" prefix to signal that this is what client, \n> git-clone here, sends; I have added also explicit \"\\n\" to mark LF\n> characters terminating lines, and put each pkt-line on separate line)\n> \n> gb> C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\\n\n> gb> C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\n\n> gb> C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\n\n> gb> C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\n\n> gb> C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\n\n> gb> C: 0000\n> gb> C: 0009done\\n\n> \n> and where server response is (again the quote from \"Git Community Book\"\n> was modified, removing here doublequotes and doubling of backslashes):\n\nThat should be fine.\n\nHere's a dummy client that works against both jgit and C Git:\n\n perl -e '\n\t$h=\"fcfcfb1fd94829c1a1704f894fc111d14770d34e\";\n\t$c=\"multi_ack side-band-64k ofs-delta\";\n sub w{$_=shift;printf \"%4.4x%s\",4+length,$_;}\n w(\"git-upload-pack /.git\");\n w(\"want $h $c\\n\");\n printf \"0000\";\n w(\"done\")\n ' | nc localhost 9418\n\nAre you sure you are flushing the IO buffers in the dummy client?\n\n-- \nShawn.\n"},{"id":"115490","messageId":"200906050924.43882.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090605004540.GU3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-05T07:24:42Z","receivedAt":"2009-06-05T07:24:42Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 5 Jun 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n> > \n> > Here we were talking about the following part of exchange: \n> > (I have added \"C:\" prefix to signal that this is what client, \n> > git-clone here, sends; I have added also explicit \"\\n\" to mark LF\n> > characters terminating lines, and put each pkt-line on separate line)\n> > \n> > gb> C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\\n\n> > gb> C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\n\n> > gb> C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\n\n> > gb> C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\n\n> > gb> C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\n\n> > gb> C: 0000\n> > gb> C: 0009done\\n\n> > \n> > and where server response is (again the quote from \"Git Community Book\"\n> > was modified, removing here doublequotes and doubling of backslashes):\n> \n> That should be fine.\n> \n> Here's a dummy client that works against both jgit and C Git:\n> \n> perl -e '\n> \t$h=\"fcfcfb1fd94829c1a1704f894fc111d14770d34e\";\n> \t$c=\"multi_ack side-band-64k ofs-delta\";\n> sub w{$_=shift;printf \"%4.4x%s\",4+length,$_;}\n> w(\"git-upload-pack /.git\");\n> w(\"want $h $c\\n\");\n> printf \"0000\";\n> w(\"done\")\n> ' | nc localhost 9418\n> \n> Are you sure you are flushing the IO buffers in the dummy client?\n\nThat is not what I meant. Perhaps I didn't explain it clear enough...\n\nThe above sequence works fine with dummy client in Perl; where it hangs\nis when client tries to wait for server response (NAK or ACK) _before_\nsending \"done\":\n\n $sock->print(pkt_line(\"want $h $c\\n\"));\n $sock->print(\"0000\");\n $sock->flush();\n\n while (!$sock->eof()) {\n my $r = $sock->read($hex4, 4); \n ...\n }\n\n $sock->print(\"0009done\\n\");\n $sock->flush();\n\nBut perhaps I did something wrong in my dummy client...\n\n\nAlso the flush \"0000\" seems to be required... but when I tried to repeat\nit using the above example it actually does not hang, but doesn't get\nPACK from git-daemon: there is something wrong in above snippet, as \nI get the same error on server whether I put \"0000\" flush line or not...\n\n c$ perl -e '\n my $h=\"c1e54552c9b35521f189db53db24cc82b5b75816\";\n my $c=\"multi_ack side-band-64k ofs-delta\";\n sub w{$_=shift;printf \"%04x%s\",4+length,$_;}\n w(\"git-upload-pack /git.git\");\n w(\"want $h $c\\n\");\n ## printf \"0000\"; # <-- commented out!\n w(\"done\");\n ' | nc localhost -v 9418\n \n s$ git daemon --export-all --verbose \\\n --base-path=/home/local/scm/ /home/local/scm/\n [12791] Connection from 127.0.0.1:42484\n [12791] Request upload-pack for '/git.git'\n fatal: git upload-pack: not our ref c1e54552c9b35521f189db53db24cc82b5b75816 multi_ack side-band-64k ofs-delta\n\n [12692] [12791] Disconnected (with error)\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115496","messageId":"m3ab4n6pc6.fsf@localhost.localdomain","threadId":"19323","inReplyTo":"200906050924.43882.jnareb@gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-05T08:45:49Z","receivedAt":"2009-06-05T08:45:49Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski writes:\n\n> Also the flush \"0000\" seems to be required... but when I tried to repeat\n> it using the above example it actually does not hang, but doesn't get\n> PACK from git-daemon: there is something wrong in above snippet, as \n> I get the same error on server whether I put \"0000\" flush line or not...\n> \n> c$ perl -e '\n> my $h=\"c1e54552c9b35521f189db53db24cc82b5b75816\";\n> my $c=\"multi_ack side-band-64k ofs-delta\";\n> sub w{$_=shift;printf \"%04x%s\",4+length,$_;}\n> w(\"git-upload-pack /git.git\");\n> w(\"want $h $c\\n\");\n> ## printf \"0000\"; # <-- commented out!\n> w(\"done\");\n> ' | nc localhost -v 9418\n> \n> s$ git daemon --export-all --verbose \\\n> --base-path=/home/local/scm/ /home/local/scm/\n> [12791] Connection from 127.0.0.1:42484\n> [12791] Request upload-pack for '/git.git'\n> fatal: git upload-pack: not our ref c1e54552c9b35521f189db53db24cc82b5b75816 multi_ack side-band-64k ofs-delta\n> \n> [12692] [12791] Disconnected (with error)\n\nWhile it works against git-clone (\"git clone git://localhost/git.git\"),\nso the problem is with the above snippet, not with git-daemon invocation.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"115657","messageId":"d411cc4a0906060933m4f27c40eoaa2b81f983adc177@mail.gmail.com","threadId":"19323","inReplyTo":"20090603165613.GJ3355@spearce.org","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-06-06T16:33:06Z","receivedAt":"2009-06-06T16:33:06Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nOn Wed, Jun 3, 2009 at 9:56 AM, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n>> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n>> > Oh, and send-pack/receive-pack protocol now has \".have\" refs [...]\n>>\n>> What are those \".have\" refs? They are not described in current version\n>> of \"Transfer Protocols\" (sub)section in \"The Community Book\". I remember\n>> that they were discussed on git mailing list, but I don't remember what\n>> they were about...\n>\n> If the remote receiving repository has alternates, the \".have\" refs are\n> the refs of the alternate repositories.  This signals to the client that\n> the server has these objects reachable, but the client isn't permitted\n> to send commands to alter these refs.\n\nCan someone help me out with the '.have' refs? What do they look\nlike? Is this the same as the '.keep' ref Jakub mentioned earlier in\none of the example server responses?\n\nI'm trying to take this whole thread and actually write an RFC style\ndocument for all of this stuff, but I'm still unclear on the .have\nportion of the conversation. Pointing me to an earlier relevant\nthread in the Git mailing list would be fine, too - it's difficult to\nsearch for '.have' usefully.\n\nThanks!\nScott\n\n>\n> Its good for a site like GitHub or repo.or.cz where cheap forks are\n> implemented by creating a repository that points to a common shared\n> base via alternates.  The \".have\" refs say that the server already\n> has everything in that common shared base, so the client doesn't\n> have to re-upload the entire project if the fork started out empty,\n> or had all refs deleted from it.\n>\n>> > In packed-refs, Junio had a hard time adding the \"peeled-refs\"\n>> > support, because the first version of the parser was so strict.\n>> > But again, somehow he managed to find a backdoor in the old parser,\n>> > and that backdoor is why that file looks the way it does today.\n>>\n>> I don't remember what that was about... Nevertheless now we have\n>> kind of 'capabilities' section in .git/packed-refs\n>\n> Sort of.  In a file format its worse than a network protocol,\n> because the file can't alter its contents based on what the\n> reader can understand.\n>\n>> Interesting... even more so that this problem of designing without\n>> extendability in mind (magic number + version) is so persistent :-(\n>\n> I know.  I think we maybe have learned the lesson.  I don't know.\n>\n> --\n> Shawn.\n>\n"},{"id":"115658","messageId":"7viqj9e0ne.fsf@alter.siamese.dyndns.org","threadId":"19323","inReplyTo":"d411cc4a0906060933m4f27c40eoaa2b81f983adc177@mail.gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-06-06T17:24:05Z","receivedAt":"2009-06-06T17:24:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Scott Chacon writes:\n\n> I'm trying to take this whole thread and actually write an RFC style\n> document for all of this stuff, but I'm still unclear on the .have\n> portion of the conversation. Pointing me to an earlier relevant\n> thread in the Git mailing list would be fine, too - it's difficult to\n> search for '.have' usefully.\n\nThe actual patch series is this.\n\n http://thread.gmane.org/gmane.comp.version-control.git/95351\n\nThe thread the patch series's cover letter refers as \"the topic discussed\nearlier\" is this.\n\n http://thread.gmane.org/gmane.comp.version-control.git/95072/focus=95256\n\nHere is how people can dig this, for people's reference.\n\n (1) Where in the code is this feature implemented?\n\n $ git grep -n -e '\\.have' -- '*.c'\n builtin-receive-pack.c:647: add_extra_ref(\".have\",...\n connect.c:87: name_len == 5 && !memcmp(\".have\", ...\n\n (2) When was it added?\n\n $ git blame -L 645,650 builtin-receive-pack.c\n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 645) ex\n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 646) ex\n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 647) \n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 648) }\n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 649) transpo\n d79796bc (Junio C Hamano 2008-09-09 01:27:10 -0700 650) free(ot\n \n (3) Go to http://news.gmane.org/gmane.comp.version-control.git/ and page\n back to the timeframe:\n"},{"id":"115659","messageId":"200906061941.16521.jnareb@gmail.com","threadId":"19323","inReplyTo":"d411cc4a0906060933m4f27c40eoaa2b81f983adc177@mail.gmail.com","subject":"Re: Request for detailed documentation of git pack protocol","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-06T17:41:15Z","receivedAt":"2009-06-06T17:41:15Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 6 Jan 2009, Scott Chacon wrote:\n> On Wed, Jun 3, 2009 at 9:56 AM, Shawn O. Pearce wrote:\n>> Jakub Narebski wrote:\n>>> On Wed, 3 Jun 2009, Shawn O. Pearce wrote:\n>>>>\n>>>> Oh, and send-pack/receive-pack protocol now has \".have\" refs [...]\n>>>\n>>> What are those \".have\" refs? They are not described in current version\n>>> of \"Transfer Protocols\" (sub)section in \"The Community Book\". I remember\n>>> that they were discussed on git mailing list, but I don't remember what\n>>> they were about...\n>>\n>> If the remote receiving repository has alternates, the \".have\" refs are\n>> the refs of the alternate repositories.  This signals to the client that\n>> the server has these objects reachable, but the client isn't permitted\n>> to send commands to alter these refs.\n> \n> Can someone help me out with the '.have' refs? What do they look\n> like? Is this the same as the '.keep' ref Jakub mentioned earlier in\n> one of the example server responses?\n\nThis was my mistake, and even more that was double mistake. It is \n'.have', not '.keep', and (as Shawn said) it can be found in response\nduring _push_ as a reply from git-receive-pack, not during fetch / clone\nas reply from git-upload-pack.\n\nIf a repository you want to push to uses alternates (e.g. was cloned\nusing --shared option, or using --reference= option),\nthen refs from repository which serves as source of alternate \n(additional) object database are shown as '.have' refs.\n\n\nCreate some repository, add some objects to it that it is not empty,\nthen clone it (locally) using e.g. \"git clone --mirror --shared\",\ndo some work on clone (for example delete one of branches), then try to\npush.\n \nI used \"ssh localhost git-receive-pack /path/to/clone.git\" as a dummy\nclient to see what response from git-receive-pack would be:\n\n 0059c0a92eb6f58c25a4c00e5e754e6de83e103231a1 .have report-status delete-refs ofs-delta\\n \n 0033efd990cb1a5f35b2b3e8b0ef0a85f43b118b8688 .have\\n\n 0033c0a92eb6f58c25a4c00e5e754e6de83e103231a1 .have\\n\n 003fefd990cb1a5f35b2b3e8b0ef0a85f43b118b8688 refs/heads/master\\n\n 0000\n\n'.have' are references in repository which given repository borrows\nobject from, i.e. which object database is in $GIT_DIR/objects/info/alternates\nfile.\n\nSidenote: here as far as I can see we do not use \"\\0\" trick... \nwhich is a bit strange (at least for me).\n\n> \n> I'm trying to take this whole thread and actually write an RFC style\n> document for all of this stuff, but I'm still unclear on the .have\n> portion of the conversation. Pointing me to an earlier relevant\n> thread in the Git mailing list would be fine, too - it's difficult to\n> search for '.have' usefully.\n\nWell, \"Transfer Protocols\" section in \"Git Community Book\" is a good\nstart. I think that it would be better to have pack protocol described\nfirst there, not necessary with the amount of detail required for \ntechnical reference documentation (format) like RFC.\n\nCurrently the \"Pushing Data\" subsection in \"Transfer Protocols\" consist\ncurrently of two short paragraphs... and that is the section where \ndescription of fake '.have' refs should go to. They do not matter and\nare not used for fetching.\n\n\nP.S. I'll try to send, as a summary of this thread (and my experiments),\nsecond round of my comments about smart protocols section of \"Transfer\nProtocols\" section soon, and most probably third round would be in the\nform of a patch to Markdown sources for this section (chapter).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115672","messageId":"200906062338.02451.jnareb@gmail.com","threadId":"19323","inReplyTo":"200906022339.08639.jnareb@gmail.com","subject":"Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-06T21:38:00Z","receivedAt":"2009-06-06T21:38:00Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"There are beginnings of description of git pack protocol in section\n\"Transfer Protocols\"[1][2] of chapter \"7. Internals and Plumbing\"\nof \"Git Community Book\" (http://book.git-scm.com).\n\n [1] http://book.git-scm.com/7_transfer_protocols.html\n [2] http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\n\nThis is second round of my comments about this item. I'd like to have\nsome more comments about git pack protocol before trying to come up\nwith formulation which is good enough to send as patch against source\nof mentioned section.\n\nThe relevant parts of above source are quoted as if they were email\nI am replying too.\n\nI have CC-ed everybody who participated in this subthread (originally\nnamed \"Re: Request for detailed documentation of git pack protocol\").\n\n....\n> ### Fetching Data with Upload Pack ###\n> \n> For the smarter protocols, fetching objects is much more efficient. A\n> socket is opened, either over ssh or over port 9418 (in the case of\n> the git:// protocol), and the git-fetch-pack(1) command on the client\n> begins communicating with a forked git-upload-pack(1) process on the\n> server.\n> \n> Then the server will tell the client which SHAs it has for each ref,\n> and the client figures out what it needs and responds with a list of\n> SHAs it wants and already has.\n\nIt would be probably more clear here to state explicitely that there\nare two lists, i.e. \"a list of SHAs it wants and a list of SHAs it\nalready has\".\n\n> \n> At this point, the server will generate a packfile with all the\n> objects that the client needs and begin streaming it down to the\n> client.\n\nThis is a bit of oversimplification. In most simple case like client\nusing git-clone to get all objects it is true that server can generate\npackfile and stream it to client after client tells a list of wanted\nSHAs. In more complicated case however there can be series of\nexchanges between client and server, with client sending sets of\ncommits it have, and server responding whether it is enough (or\nperhaps this line of commits is uninteresting)... and only then\narriving at list of objects to send in a packfile.\n\n> \n> Let's look at an example.\n\nI think that before example we should have short description (sketch)\nof the whole exchange; for example the one taken from\n'Documentation/technical/pack-protocol.txt':\n\nupload-pack (S) | fetch/clone-pack (C) protocol:\n\n # Tell the puller what commits we have and what their names are\n S: SHA1 name\n S: ...\n S: SHA1 name\n S: # flush -- it's your turn\n # Tell the pusher what commits we want, and what we have\n C: want name\n C: ..\n C: want name\n C: have SHA1\n C: have SHA1\n C: ...\n C: # flush -- occasionally ask \"had enough?\"\n S: NAK\n C: have SHA1\n C: ...\n C: have SHA1\n S: ACK\n C: done\n S: XXXXXXX -- packfile contents.\n\n\n> \n> The client connects and sends the request header. The clone command\n> \n> \t$ git clone git://myserver.com/project.git\n> \n> produces the following request:\n> \n> \t0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n\nAlthough fetching via SSH protocol is, I guess, much more rare than\nfetching via anonymous unauthenticated git:// protocol, it _might_ be\ngood idea to tell there that fetching via SSH differs from above\nsequence that instead of opening TCP connection to port 9418 and\nsending above packet, and later reading from and writing to socket,\n\"git clone ssh://myserver.com/srv/git/project.git\" calls\n\n\tssh myserver.com git-upload-pack /srv/git/project.git\n\nand later reads from standard output of the above command, and writes\nto standard input of above command.\n\nThe rest of exchange is _identical_ for git:// and for ssh:// (and\nI guess also for file:// pseudoprotocol).\n\n> \n> The first four bytes contain the hex length of the line (including 4\n> byte line length and trailing newline if present). Following are the\n> command and arguments. This is followed by a null byte and then the\n> host information. The request is terminated by a null byte.\n\nI think it would be better to describe packet (chunk) format, called\npkt-line in git, separately from describing the contents of above\npacket; either first pkt-line then command, or first command then\npkt-line. Otherwise we would be left with describing pkt-line format\nmany times, as it is done in current version of this chapter.\n\n\nIn git clients communicates with server using a packetized stream,\nwhere each line (packet, chunk) is preceded by its length (including\nthe header) as a 4-byte hex number. A length of 'zero', i.e. packet\n\"0000\" has a special meaning: it means end of stream / flush\nconnection. The \"# flush ...\" in description of client--server\nexchange above is done using exactly \"0000\" packet. \n\nFootnote: this format somewhat reminds / resembles 'chunked' transfer\nencoding used in HTTP[1], although there are differences.\n http://en.wikipedia.org/wiki/Chunked_transfer_encoding\n\n> \n> The request is processed and turned into a call to git-upload-pack:\n> \n> \t$ git-upload-pack /path/to/repos/project.git\n\nThis is alternate place where we could tell about fetching via ssh://\n\nWe probably should tell where /path/to/repos that /project.git is\nprefixed with comes from; it is from --base-path=/path/to/repos\nargument to git-daemon (a sort of \"GIT root\").\n\nBTW. (this is just a very minor nit) shouldn't we use FHS compliant\npath, i.e. \"/srv/git\" instead of \"/path/to/repos\" (and follow RFC in\nusing \"example.com\" in place of \"myserver.com\")?\n\n> \n> This immediately returns information of the repo:\n> \n> \t007c74730d410fcb6603ace96f1dc55ea6196122532d HEAD\\\\000multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag\\\\n\n> \t003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\\\\n\n> \t003d5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/dist\\\\n\n> \t003e7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 refs/heads/local\\\\n\n> \t003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\\\\n\n> \t0000\n\nI have added explicit LF terminators in the form of \"\\\\n\" (which would\nrender as \"\\n\"), mainly because \"0000\" flush packed _doesn't_ have it.\nAlso I have added \"include-tag\", as modern git installations provide\nthis capability.\n\nHere is a dilemma: currently example output is provided almost exactly\nas-is, only indented and with some quoting/escaping (\\\\000 or \\\\0 for\nNUL character, \\\\n for LF, later \\\\001 and \\\\002 for 0x01 and 0x02\nbytes). To know if given example output is what client sends or what\nserver outputs, you have to read the narrative. Alternate solution\nwould be to use \"C: \" and \"S: \" prefixing (perhaps with some extra\nformat to make it more clear that it is not part of data), used in\npack-protocol.txt technical documentation, and proposed for describing\nnetwork protocols by some RFC (I don't remember which, unfortunately).\nWhich one to choose?\n\n\nWe would want, at some point, describe that first line of first\nresponse from server contains 'stuffed' behind \"\\0\" (NUL) space\nseparated list of capabilities our server supports. Those\ncapabilities would have to be described somewhere: as a sidebar, \nor in a separate subsection, or in an appendix.\n\nBelow there is (for completeness) list of git-upload-pack\ncapabilities, with short description of each:\n\n * multi_ack (for historical reasons not multi-ack)\n\n It allows the server to return \"ACK $SHA1 continue\" as soon as it\n finds a commit that it can use as a common base, between the\n client's wants and the client's have set.\n\n By sending this early, the server can potentially head off the\n client from walking any further down that particular branch of the\n client's repository history.\n \n See the thread for more details (posts by Shawn O. Pearce and by\n Junio C Hamano).\n\n * thin-pack\n\n Server can send thin packs, i.e. packs which do not contain base\n elements for some delta chains, if those base elements are\n available on client side. Client has thin-pack capability when it\n understand how to \"thicken\" them adding required delta bases,\n making those packfiles independent.\n\n Of course it doesn't make sense for client to use (request) this\n capability for git-clone... But if the client does request it (and\n I think modern clients actually do request it, even on initial\n clone case) the server won't produce a thin pack. Why? There is no\n common base, so there is no uninteresting set to omit from the\n pack. :-)\n \n * side-band \n * side-band-64k \n\n This means that server can send, and client understand multiplexed\n (muxed) progress reports and error info interleaved with the\n packfile itself.\n\n These two options are mutually exclusive. A client should ask for\n only one of them, and a modern client always favors side-band-64k.\n If client ask for both, server uses side-band-64k.\n\n Older side-band allows only up to 1000 bytes per packet.\n\n * ofs-delta \n\n Server can send, and client understand PACKv2 with delta refering\n to its base by position in pack rather than by SHA-1. Both can\n send/read OBJ_OFS_DELTA, aka type 6 in a pack file.\n\n * shallow \n\n Server can send shallow clone (git clone --depth ...).\n\n * no-progress\n\n Client should use it if it was started with \"git clone -q\" or\n something, and doesn't want that side brand 2. We still want\n sideband 1 with actual data (packfile), and sideband 3 with error\n messages.\n\n * include-tag\n\n If we pack an object to the client, and a tag points exactly at\n that object, we pack the tag too. In general this allows a client\n to get all new tags when it fetches a branch, in a single network\n connection, instead of two (separate connection for tags).\n\n This capability is not to be used when client was called with\n '--no-tags'.\n\n> \n> Each line starts with a four byte line length declaration in hex. The\n> section is terminated by a line length declaration of 0000.\n\nThis repetition would not be necessary if pkt-line format had its own\ndescription somewhere before. We would probably still want to remind\nthe reader that \"0000\" line length declaration means 'flush'.\n\n> \n> This is sent back to the client verbatim. \n\nHmmm... \"sent back ... verbatim\"? I wonder what did you want to say\nhere...\n\n> The client responds with another request:\n> \n> \t0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\\\\n\n> \t0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\\\n\n> \t0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\\\n\n> \t0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\\\n\n> \t0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\\\n\n> \t0000\n> 0009done\\\\n\n\nHere again I added explicit LF terminator, and split off \"0000\" flush\npacket in separate line, to make this request (well, two requests)\nmore clear.\n\nThe first line of this request contains capabilities client wants to\nuse. It should be some subset of capabilities server supports.\n\n> \n> The is sent to the open git-upload-pack process which then streams out\n> the final response:\n\n\"_The_ is send\"?\n\nI would remove quotes around lines of server response below, but would\nleave explicit \\n for LF, and \\\\001 and \\\\002 for bytes 0x01 and 0x02\ndenoting channel.\n\n> \n> \t\"0008NAK\\n\"\n\nThis NAK means that server did not found [closed] set of common\nancestors. It is response to \"0000\" flush line (\"had enough?\" line)\nfrom client. As the example is about git-clone, and client doesn't\n_have_ any commits to show server as candidates for common ancestors\n(calculation), it replies with \"done\" to get pack.\n\n> \t\"0023\\\\002Counting objects: 2797, done.\\n\"\n\nThis is a bit untypical example, as for larger repositories like Linux\nkernel or even git repository, usually you would have much more\nobjects, and actually object enumeration would take more time. You\nwould see many\n\n\t\"0020\\\\002Counting objects: 10662 \\r\"\n\t\"0020\\\\002Counting objects: 22318 \\r\"\n\t\"0020\\\\002Counting objects: 29506 \\r\"\n\npackets before\n\n \t\"0023\\\\002Counting objects: 65058, done.\\n\"\n\n> \t\"002b\\\\002Compressing objects: 0% (1/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 1% (12/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 2% (24/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 3% (36/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 4% (48/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 5% (59/1177) \\r\"\n> \t\"002c\\\\002Compressing objects: 6% (71/1177) \\r\"\n> \t\"0053\\\\002Compressing objects: 7% (83/1177) \\rCompressing objects: 8% (95/1177) \\r\"\n> \t...\n> \t\"005b\\\\002Compressing objects: 100% (1177/1177) \\rCompressing objects: 100% (1177/1177), done.\\n\"\n\nSidenote: the reason why there is sometimes more than one line send in\na single packet / single pkt-line is buffering between git-pack-objects\nwhich produces those messages to pipe, and git-upload-pack which reads\nthem and sends them to client. If pack-objects can write two messages\ninto the pipe buffer before upload-pack is woken to read them out,\nupload-pack might find two (or more) messages ready to read without\nblocking. These get bundled into a single packet, because, why not,\nits easier to code it that way.\n\nHere or a little later we probably should explain (even though it is\nfairly obvious), that final response from server is (here) in pkt-line\nwith sideband format, where first byte of data denotes channel\n(stream) number: 1 for data, 2 for progress info, 3 for fatal errors.\n\n> \t\"2004\\\\001PACK\\\\000\\\\000\\\\000\\\\002\\\\000\\\\000\\n\\\\355\\\\225\\\\017x\\\\234\\\\235\\\\216K\\n\\\\302\"...\n> \t\"2005\\\\001\\\\360\\\\204{\\\\225\\\\376\\\\330\\\\345]z\\226\\273\"...\n\nHere I think it would be enough to show only the fragment which is\npackfile signature...\n\n> \t...\n> \t\"0037\\\\002Total 2797 (delta 1799), reused 2360 (delta 1529)\\n\"\n> \t...\n> \t\"<\\\\276\\\\255L\\\\273s\\\\005\\\\001w0006\\\\001[0000\"\n\nThis line is I think is broken in wrong place. It is the tail\nend of some packet (each packed begins with 4 characters wide 0-padded\nlength of chunk as hex number; \"<\\\\276\\\\255L\" does not match 4HEXDIG),\nfollowed by \"0000\" 'flush' packet (here it signals end of stream).\n\n> \t\n> See the Packfile chapter previously for the actual format of the\n> packfile data in the response.\n> \n> \n....\n-- \nJakub Narebski\nPoland\n"},{"id":"115673","messageId":"d411cc4a0906061458g494d80dbwe3a5358edfd1d49e@mail.gmail.com","threadId":"19323","inReplyTo":"200906062338.02451.jnareb@gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-06-06T21:58:00Z","receivedAt":"2009-06-06T21:58:00Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nOn Sat, Jun 6, 2009 at 2:38 PM, Jakub Narebski wrote:\n> There are beginnings of description of git pack protocol in section\n> \"Transfer Protocols\"[1][2] of chapter \"7. Internals and Plumbing\"\n> of \"Git Community Book\" (http://book.git-scm.com).\n>\n>  [1] http://book.git-scm.com/7_transfer_protocols.html\n>  [2] http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\n>\n> This is second round of my comments about this item. I'd like to have\n> some more comments about git pack protocol before trying to come up\n> with formulation which is good enough to send as patch against source\n> of mentioned section.\n>\n\nI can certainly fix up this chapter with these comments - I understand\nthe protocol a bit better now than I did when I originally wrote this.\n\nIn addition to that, I started taking a shot at putting together an\nRFC formatted documentation of this protocol as was requested. I may\nhave _way_ missed the mark on what you were looking for originally,\nit's hard to say, not having read a lot of RFC documents - I probably\nended up writing in a more bookish format rather than a technical\nspec, but whatever - maybe you'll find it helpful or can fix it up to\nmore what you were expecting. I'm not done with it - some of it is\nstill basically unformatted comments from this previous thread, but at\nleast it's laid out roughly how I thought it might be useful and I\nhave fleshed out a lot of it. You can find the RFC text output\ndocument here:\n\nhttp://git-scm.com/gitserver.txt\n\nAnd the xml doc I generated it from here:\n\nhttp://github.com/schacon/gitserver-rfc\n\nPerhaps if we're going to spend time getting this all correct, we\nshould get a standalone technical doc all agreed upon, then I can\nrelatively easily extract what's needed into that chapter of the\nCommunity book.\n\nThoughts?\n\nScott\n\n> The relevant parts of above source are quoted as if they were email\n> I am replying too.\n>\n> I have CC-ed everybody who participated in this subthread (originally\n> named \"Re: Request for detailed documentation of git pack protocol\").\n>\n> ....\n>> ### Fetching Data with Upload Pack ###\n>>\n>> For the smarter protocols, fetching objects is much more efficient.  A\n>> socket is opened, either over ssh or over port 9418 (in the case of\n>> the git:// protocol), and the git-fetch-pack(1) command on the client\n>> begins communicating with a forked git-upload-pack(1) process on the\n>> server.\n>>\n>> Then the server will tell the client which SHAs it has for each ref,\n>> and the client figures out what it needs and responds with a list of\n>> SHAs it wants and already has.\n>\n> It would be probably more clear here to state explicitely that there\n> are two lists, i.e. \"a list of SHAs it wants and a list of SHAs it\n> already has\".\n>\n>>\n>> At this point, the server will generate a packfile with all the\n>> objects that the client needs and begin streaming it down to the\n>> client.\n>\n> This is a bit of oversimplification.  In most simple case like client\n> using git-clone to get all objects it is true that server can generate\n> packfile and stream it to client after client tells a list of wanted\n> SHAs.  In more complicated case however there can be series of\n> exchanges between client and server, with client sending sets of\n> commits it have, and server responding whether it is enough (or\n> perhaps this line of commits is uninteresting)... and only then\n> arriving at list of objects to send in a packfile.\n>\n>>\n>> Let's look at an example.\n>\n> I think that before example we should have short description (sketch)\n> of the whole exchange; for example the one taken from\n> 'Documentation/technical/pack-protocol.txt':\n>\n> upload-pack (S) | fetch/clone-pack (C) protocol:\n>\n>  # Tell the puller what commits we have and what their names are\n>  S: SHA1 name\n>  S: ...\n>  S: SHA1 name\n>  S: # flush -- it's your turn\n>  # Tell the pusher what commits we want, and what we have\n>  C: want name\n>  C: ..\n>  C: want name\n>  C: have SHA1\n>  C: have SHA1\n>  C: ...\n>  C: # flush -- occasionally ask \"had enough?\"\n>  S: NAK\n>  C: have SHA1\n>  C: ...\n>  C: have SHA1\n>  S: ACK\n>  C: done\n>  S: XXXXXXX -- packfile contents.\n>\n>\n>>\n>> The client connects and sends the request header. The clone command\n>>\n>>       $ git clone git://myserver.com/project.git\n>>\n>> produces the following request:\n>>\n>>       0032git-upload-pack /project.git\\\\000host=myserver.com\\\\000\n>\n> Although fetching via SSH protocol is, I guess, much more rare than\n> fetching via anonymous unauthenticated git:// protocol, it _might_ be\n> good idea to tell there that fetching via SSH differs from above\n> sequence that instead of opening TCP connection to port 9418 and\n> sending above packet, and later reading from and writing to socket,\n> \"git clone ssh://myserver.com/srv/git/project.git\" calls\n>\n>        ssh myserver.com git-upload-pack /srv/git/project.git\n>\n> and later reads from standard output of the above command, and writes\n> to standard input of above command.\n>\n> The rest of exchange is _identical_ for git:// and for ssh:// (and\n> I guess also for file:// pseudoprotocol).\n>\n>>\n>> The first four bytes contain the hex length of the line (including 4\n>> byte line length and trailing newline if present). Following are the\n>> command and arguments. This is followed by a null byte and then the\n>> host information. The request is terminated by a null byte.\n>\n> I think it would be better to describe packet (chunk) format, called\n> pkt-line in git, separately from describing the contents of above\n> packet; either first pkt-line then command, or first command then\n> pkt-line.  Otherwise we would be left with describing pkt-line format\n> many times, as it is done in current version of this chapter.\n>\n>\n> In git clients communicates with server using a packetized stream,\n> where each line (packet, chunk) is preceded by its length (including\n> the header) as a 4-byte hex number.  A length of 'zero', i.e. packet\n> \"0000\" has a special meaning: it means end of stream / flush\n> connection.  The \"# flush ...\" in description of client--server\n> exchange above is done using exactly \"0000\" packet.\n>\n> Footnote: this format somewhat reminds / resembles 'chunked' transfer\n> encoding used in HTTP[1], although there are differences.\n>  http://en.wikipedia.org/wiki/Chunked_transfer_encoding\n>\n>>\n>> The request is processed and turned into a call to git-upload-pack:\n>>\n>>       $ git-upload-pack /path/to/repos/project.git\n>\n> This is alternate place where we could tell about fetching via ssh://\n>\n> We probably should tell where /path/to/repos that /project.git is\n> prefixed with comes from; it is from --base-path=/path/to/repos\n> argument to git-daemon (a sort of \"GIT root\").\n>\n> BTW. (this is just a very minor nit) shouldn't we use FHS compliant\n> path, i.e. \"/srv/git\" instead of \"/path/to/repos\" (and follow RFC in\n> using \"example.com\" in place of \"myserver.com\")?\n>\n>>\n>> This immediately returns information of the repo:\n>>\n>>       007c74730d410fcb6603ace96f1dc55ea6196122532d HEAD\\\\000multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag\\\\n\n>>       003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\\\\n\n>>       003d5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/dist\\\\n\n>>       003e7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 refs/heads/local\\\\n\n>>       003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\\\\n\n>>       0000\n>\n> I have added explicit LF terminators in the form of \"\\\\n\" (which would\n> render as \"\\n\"), mainly because \"0000\" flush packed _doesn't_ have it.\n> Also I have added \"include-tag\", as modern git installations provide\n> this capability.\n>\n> Here is a dilemma: currently example output is provided almost exactly\n> as-is, only indented and with some quoting/escaping (\\\\000 or \\\\0 for\n> NUL character, \\\\n for LF, later \\\\001 and \\\\002 for 0x01 and 0x02\n> bytes).  To know if given example output is what client sends or what\n> server outputs, you have to read the narrative.  Alternate solution\n> would be to use \"C: \" and \"S: \" prefixing (perhaps with some extra\n> format to make it more clear that it is not part of data), used in\n> pack-protocol.txt technical documentation, and proposed for describing\n> network protocols by some RFC (I don't remember which, unfortunately).\n> Which one to choose?\n>\n>\n> We would want, at some point, describe that first line of first\n> response from server contains 'stuffed' behind \"\\0\" (NUL) space\n> separated list of capabilities our server supports.  Those\n> capabilities would have to be described somewhere: as a sidebar,\n> or in a separate subsection, or in an appendix.\n>\n> Below there is (for completeness) list of git-upload-pack\n> capabilities, with short description of each:\n>\n>  * multi_ack (for historical reasons not multi-ack)\n>\n>   It allows the server to return \"ACK $SHA1 continue\" as soon as it\n>   finds a commit that it can use as a common base, between the\n>   client's wants and the client's have set.\n>\n>   By sending this early, the server can potentially head off the\n>   client from walking any further down that particular branch of the\n>   client's repository history.\n>\n>   See the thread for more details (posts by Shawn O. Pearce and by\n>   Junio C Hamano).\n>\n>  * thin-pack\n>\n>   Server can send thin packs, i.e. packs which do not contain base\n>   elements for some delta chains, if those base elements are\n>   available on client side.  Client has thin-pack capability when it\n>   understand how to \"thicken\" them adding required delta bases,\n>   making those packfiles independent.\n>\n>   Of course it doesn't make sense for client to use (request) this\n>   capability for git-clone... But if the client does request it (and\n>   I think modern clients actually do request it, even on initial\n>   clone case) the server won't produce a thin pack. Why?  There is no\n>   common base, so there is no uninteresting set to omit from the\n>   pack.  :-)\n>\n>  * side-band\n>  * side-band-64k\n>\n>   This means that server can send, and client understand multiplexed\n>   (muxed) progress reports and error info interleaved with the\n>   packfile itself.\n>\n>   These two options are mutually exclusive. A client should ask for\n>   only one of them, and a modern client always favors side-band-64k.\n>   If client ask for both, server uses side-band-64k.\n>\n>   Older side-band allows only up to 1000 bytes per packet.\n>\n>  * ofs-delta\n>\n>   Server can send, and client understand PACKv2 with delta refering\n>   to its base by position in pack rather than by SHA-1.  Both can\n>   send/read OBJ_OFS_DELTA, aka type 6 in a pack file.\n>\n>  * shallow\n>\n>   Server can send shallow clone (git clone --depth ...).\n>\n>  * no-progress\n>\n>   Client should use it if it was started with \"git clone -q\" or\n>   something, and doesn't want that side brand 2.  We still want\n>   sideband 1 with actual data (packfile), and sideband 3 with error\n>   messages.\n>\n>  * include-tag\n>\n>   If we pack an object to the client, and a tag points exactly at\n>   that object, we pack the tag too.  In general this allows a client\n>   to get all new tags when it fetches a branch, in a single network\n>   connection, instead of two (separate connection for tags).\n>\n>   This capability is not to be used when client was called with\n>   '--no-tags'.\n>\n>>\n>> Each line starts with a four byte line length declaration in hex. The\n>> section is terminated by a line length declaration of 0000.\n>\n> This repetition would not be necessary if pkt-line format had its own\n> description somewhere before.  We would probably still want to remind\n> the reader that \"0000\" line length declaration means 'flush'.\n>\n>>\n>> This is sent back to the client verbatim.\n>\n> Hmmm... \"sent back ... verbatim\"? I wonder what did you want to say\n> here...\n>\n>> The client responds with another request:\n>>\n>>       0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\\\\n\n>>       0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\\\n\n>>       0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\\\n\n>>       0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\\\n\n>>       0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\\\n\n>>       0000\n>>       0009done\\\\n\n>\n> Here again I added explicit LF terminator, and split off \"0000\" flush\n> packet in separate line, to make this request (well, two requests)\n> more clear.\n>\n> The first line of this request contains capabilities client wants to\n> use.  It should be some subset of capabilities server supports.\n>\n>>\n>> The is sent to the open git-upload-pack process which then streams out\n>> the final response:\n>\n> \"_The_ is send\"?\n>\n> I would remove quotes around lines of server response below, but would\n> leave explicit \\n for LF, and \\\\001 and \\\\002 for bytes 0x01 and 0x02\n> denoting channel.\n>\n>>\n>>       \"0008NAK\\n\"\n>\n> This NAK means that server did not found [closed] set of common\n> ancestors. It is response to \"0000\" flush line (\"had enough?\" line)\n> from client. As the example is about git-clone, and client doesn't\n> _have_ any commits to show server as candidates for common ancestors\n> (calculation), it replies with \"done\" to get pack.\n>\n>>       \"0023\\\\002Counting objects: 2797, done.\\n\"\n>\n> This is a bit untypical example, as for larger repositories like Linux\n> kernel or even git repository, usually you would have much more\n> objects, and actually object enumeration would take more time.  You\n> would see many\n>\n>        \"0020\\\\002Counting objects: 10662   \\r\"\n>        \"0020\\\\002Counting objects: 22318   \\r\"\n>        \"0020\\\\002Counting objects: 29506   \\r\"\n>\n> packets before\n>\n>        \"0023\\\\002Counting objects: 65058, done.\\n\"\n>\n>>       \"002b\\\\002Compressing objects:   0% (1/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   1% (12/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   2% (24/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   3% (36/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   4% (48/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   5% (59/1177)   \\r\"\n>>       \"002c\\\\002Compressing objects:   6% (71/1177)   \\r\"\n>>       \"0053\\\\002Compressing objects:   7% (83/1177)   \\rCompressing objects:   8% (95/1177)   \\r\"\n>>       ...\n>>       \"005b\\\\002Compressing objects: 100% (1177/1177)   \\rCompressing objects: 100% (1177/1177), done.\\n\"\n>\n> Sidenote: the reason why there is sometimes more than one line send in\n> a single packet / single pkt-line is buffering between git-pack-objects\n> which produces those messages to pipe, and git-upload-pack which reads\n> them and sends them to client.  If pack-objects can write two messages\n> into the pipe buffer before upload-pack is woken to read them out,\n> upload-pack might find two (or more) messages ready to read without\n> blocking.  These get bundled into a single packet, because, why not,\n> its easier to code it that way.\n>\n> Here or a little later we probably should explain (even though it is\n> fairly obvious), that final response from server is (here) in pkt-line\n> with sideband format, where first byte of data denotes channel\n> (stream) number: 1 for data, 2 for progress info, 3 for fatal errors.\n>\n>>       \"2004\\\\001PACK\\\\000\\\\000\\\\000\\\\002\\\\000\\\\000\\n\\\\355\\\\225\\\\017x\\\\234\\\\235\\\\216K\\n\\\\302\"...\n>>       \"2005\\\\001\\\\360\\\\204{\\\\225\\\\376\\\\330\\\\345]z\\226\\273\"...\n>\n> Here I think it would be enough to show only the fragment which is\n> packfile signature...\n>\n>>       ...\n>>       \"0037\\\\002Total 2797 (delta 1799), reused 2360 (delta 1529)\\n\"\n>>       ...\n>>       \"<\\\\276\\\\255L\\\\273s\\\\005\\\\001w0006\\\\001[0000\"\n>\n> This line is I think is broken in wrong place.  It is the tail\n> end of some packet (each packed begins with 4 characters wide 0-padded\n> length of chunk as hex number; \"<\\\\276\\\\255L\" does not match 4HEXDIG),\n> followed by \"0000\" 'flush' packet (here it signals end of stream).\n>\n>>\n>> See the Packfile chapter previously for the actual format of the\n>> packfile data in the response.\n>>\n>>\n> ....\n> --\n> Jakub Narebski\n> Poland\n>\n"},{"id":"115689","messageId":"200906071021.08922.jnareb@gmail.com","threadId":"19323","inReplyTo":"d411cc4a0906061458g494d80dbwe3a5358edfd1d49e@mail.gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-07T08:21:07Z","receivedAt":"2009-06-07T08:21:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 6 June 2009, Scott Chacon wrote:\n\n> In addition to that, I started taking a shot at putting together an\n> RFC formatted documentation of this protocol as was requested. I may\n> have _way_ missed the mark on what you were looking for originally,\n> it's hard to say, not having read a lot of RFC documents - I probably\n> ended up writing in a more bookish format rather than a technical\n> spec, but whatever - maybe you'll find it helpful or can fix it up to\n> more what you were expecting. I'm not done with it - some of it is\n> still basically unformatted comments from this previous thread, but at\n> least it's laid out roughly how I thought it might be useful and I\n> have fleshed out a lot of it. You can find the RFC text output\n> document here:\n> \n> http://git-scm.com/gitserver.txt\n\n[...]\n> Thoughts?\n\nThose are only preliminary thoughts; more detailed analysis is to follow \n(hopefully).\n\nUsually RFC documents refer to RFC 2119 (Key words for use in RFCs to \nIndicate Requirement Levels) for definitions of words such as MUST, \nSHOULD, MAY in the following way:\n\n The key words \"MUST\", \"MUST NOT\", \"REQUIRED\", \"SHALL\", \"SHALL\n NOT\", \"SHOULD\", \"SHOULD NOT\", \"RECOMMENDED\", \"MAY\", and\n \"OPTIONAL\" in this document are to be interpreted as described in\n [RFC 2119][1].\n\n [1]: http://tools.ietf.org/html/rfc2119\n\nDefinitions are done using RFC 5234 (Augmented BNF for Syntax \nSpecifications: ABNF), referring to it for example in the following \nway:\n\n All the mechanisms specified in this document are described in both\n prose and an augmented Backus-Naur form (ABNF). It is described in\n detail in [RFC 4234][2].\n\n [2]: http://tools.ietf.org/html/rfc5234\n\n\nThe description of pkt-line and pkt-line-sb formats is wrong: length\nincludes the header. It is IMHO more natural to define it from \ngenerality to detail, and not in reverse direction; instead of this:\n\n pkt-length = 4HEXDIGIT ; length of pkt-payload\n pkt-line = pkt-length pkt-payload [ LF / CR ]\n\nfor example like this:\n\n pkt-line = pkt-length pkt-payload [ LF ]\n pkt-length = 4HEXDIGIT ; length of pkt-line (including pkt-length)\n\nBy the way, we probably accept any terminator, but I'd rather standarize \non LF (\"\\n\").\n\n\nIn description of sideband:\n\n> When a sideband is used, 2 means \"progress messages, most likely\n> suitable for stderr\". 1 means \"pack data\". 3 means \"fatal error\n> message, and we're dead now\". No other channels are used or valid.\n\nit is true that no other channels are used, but it is not true that \nother channels are invalid. If they are not supported by client, there \nare simply dropped. This opens possibility of future extension. I guess \nthat channel 0 is invalid, because it would be understood as _input_ \nchannel (for sending data from client to server), though.\n\nPlease correct me if I am wrong here...\n\n\nP.S. Could you please try to not quote large fragments of email which\nyou do not refer to in your reply, and which are not relevant to given \npost, like the long quoting at the end of your email without any word \nfrom you? Thanks in advance.\n-- \nJakub Narebski\nPoland\n"},{"id":"115725","messageId":"20090607200617.GA16497@spearce.org","threadId":"19323","inReplyTo":"200906062338.02451.jnareb@gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-07T20:06:17Z","receivedAt":"2009-06-07T20:06:17Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> There are beginnings of description of git pack protocol in section\n> \"Transfer Protocols\"[1][2] of chapter \"7. Internals and Plumbing\"\n> of \"Git Community Book\" (http://book.git-scm.com).\n\nI'm going to try to clip unrelevant context... but I apologize if\nI still quoted too much, there's a lot of text here.\n\n> > ### Fetching Data with Upload Pack ###\n...\n> Although fetching via SSH protocol is, I guess, much more rare than\n> fetching via anonymous unauthenticated git:// protocol,\n\nActually, fetching via SSH might be quite common, think about all of\nthose companies using Git internally... they are running something\nlike Gitosis or Gerrit Code Review, both of which support SSH only\naccess to the hosted repositories.\n\n> it _might_ be\n> good idea to tell there that fetching via SSH differs from above\n> sequence that instead of opening TCP connection to port 9418 and\n> sending above packet, and later reading from and writing to socket,\n> \"git clone ssh://myserver.com/srv/git/project.git\" calls\n> \n> \tssh myserver.com git-upload-pack /srv/git/project.git\n> \n> and later reads from standard output of the above command, and writes\n> to standard input of above command.\n\nYes, this should be mentioned. We actually should document in\nthe protocol specifiction how we fork SSH, and what the SSH server\nshould then be presenting as the command line.\n\nI've run into problems with hosting sites like GitHub and Gitoriuous\nnot correctly honoring some ssh invokes, because they use the forced\ncommand execution model and were handling only one case that could\nbe presented to them.\n\n> The rest of exchange is _identical_ for git:// and for ssh:// (and\n> I guess also for file:// pseudoprotocol).\n\nYes, the file:// pseduoprotocol works by forking a child to run the\n`git-upload-pack /srv/git/project.git` executable and runs a pair\nof pipes between them, just like ssh:// does when it spawns off\nthe ssh client process.\n\n \n> I think it would be better to describe packet (chunk) format, called\n> pkt-line in git, separately from describing the contents of above\n> packet; either first pkt-line then command, or first command then\n> pkt-line.\n\npkt-line is a basic building block, describe it early, before we\ndescribe anything else.\n\n> Footnote: this format somewhat reminds / resembles 'chunked' transfer\n> encoding used in HTTP[1], although there are differences.\n> http://en.wikipedia.org/wiki/Chunked_transfer_encoding\n\nThis is not worth mentioning. pkt-line is different enough that\nit may just confuse the reader.\n \n> Below there is (for completeness) list of git-upload-pack\n> capabilities, with short description of each:\n> \n> * multi_ack (for historical reasons not multi-ack)\n...\n> See the thread for more details (posts by Shawn O. Pearce and by\n> Junio C Hamano).\n\nThis really needs a diagram example, like the one I drew, to\nexplain the concept. Its really hard to grasp from just reading\nthat paragraph what that implies, especially if you are implementing\na client or a server.\n\n \n> * no-progress\n> \n> Client should use it if it was started with \"git clone -q\" or\n> something, and doesn't want that side brand 2. We still want\n\ntypo, should be \"... side band 2.\" :-)\n\n> sideband 1 with actual data (packfile), and sideband 3 with error\n> messages.\n\nAlso, this capability really only makes sense if side-band or\nside-band-64k was requested. IOW, a sane client wouldn't ask\nfor this if it doesn't support side-band.\n\n-- \nShawn.\n"},{"id":"115726","messageId":"20090607201324.GB16497@spearce.org","threadId":"19323","inReplyTo":"200906071021.08922.jnareb@gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-07T20:13:24Z","receivedAt":"2009-06-07T20:13:24Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> In description of sideband:\n> \n> > When a sideband is used, 2 means \"progress messages, most likely\n> > suitable for stderr\". 1 means \"pack data\". 3 means \"fatal error\n> > message, and we're dead now\". No other channels are used or valid.\n> \n> it is true that no other channels are used, but it is not true that \n> other channels are invalid. If they are not supported by client, there \n> are simply dropped. This opens possibility of future extension. I guess \n> that channel 0 is invalid, because it would be understood as _input_ \n> channel (for sending data from client to server), though.\n> \n> Please correct me if I am wrong here...\n\nAn implementation reading a muxed stream SHOULD fail fast if it\nencounters a channel number it doesn't understand.\n\nJGit already fails fast with an error if it gets anything not in 1-3.\nC Git already fails fast with an error as well.\n\nAn implementation writing a muxed stream shouldn't produce a channel\nnumber unless it knows the reader can support it.\n\nTo add a new channel number to the supported set, a new capability\nshould be introduced to the protocol, and enabled if both sides\nhave agreed to support it.\n\nCurrently, stream 0 and stream 4-255 are undefined. That is,\nany new capability could claim that stream and start to use it,\nif it needed to.\n\nI think the primary Git contributors would prefer to see new channels\nin the 4-255 range, as then 0 can continue to stay invalid... aka\n\"not true\" in C. Like in the pack type codes, we might want to save\n0 for the day when all 1-255 are filled and we need to expand the\nchannel number range into 2 bytes. But even then, we could just\ndo a new side-band-64kv2 capability or something. :-)\n \n-- \nShawn.\n"},{"id":"115733","messageId":"20090607204343.GC16497@spearce.org","threadId":"19323","inReplyTo":"d411cc4a0906061458g494d80dbwe3a5358edfd1d49e@mail.gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-07T20:43:43Z","receivedAt":"2009-06-07T20:43:43Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Scott Chacon wrote:\n> In addition to that, I started taking a shot at putting together an\n> RFC formatted documentation of this protocol as was requested.\n...\n> http://git-scm.com/gitserver.txt\n\nSSH is described by RFC 4251 and RFC 4254. Reference it when you\nmention it.\n\nSection 2.2.3 Commit is missing spaces after the parent, author,\ncommitter, encoding headers:\n\n> parent = \"parent\" + sha + \\n\n> userinfo = NAME TIME\n> author = \"author\" + userinfo + \\n\n> committer = \"committer\" + userinfo + \\n\n> encoding = \"encoding\" + encoding + \\n\n\n2.2.4. Tag, same problem.\n\n> At the end of the\n> packfile is a 20-byte SHA1 sum of all the shas in that packfile.\n\nNo. The SHA-1 checksum on the footer of the pack is over all of\nthe preceeding bytes of the pack.\n\n> (B << 4) & A bytes when expanded\n\nNo. (B << 4) | A bytes when expanded.\n\n> [1 byte] | 1 | type (3) | size A (4) | |- object #3 header\n> +-------------------------------+ |\n> [1 byte] | 0 | size data B (7) | |\n> +-------------------------------+ |\n> [1 byte] | 0 | size data C (7) | |\n> +-------------------------------+ -+\n> | compressed object data | (C << 11) & (B << 4) & A\n> | | bytes when expanded\n\nThe B byte has the high bit set (1). And the length is\n(C << 11) | (B << 4) | A.\n\nAlso, I found reading that difficult, and it doesn't mention the\nOBJ_REF_DELTA or OBJ_OFS_DELTA cases.\n\nYou also need to note that the version number in the file header\nis currently '2', as described by this RFC.\n\n> Finally, the trailer records 20-byte SHA1 checksum of the rest of the\n> file.\n\nLike I said above, its the preceeding bytes of the pack.\n\nSection 4.2 Git Protocol, explain the git:// URI first, and then\nhow a client splits that into the request, and then how it formats\nthe request. Don't forget to include an example with a non-standard\nport number.\n\nAlso document what the standard port number is.\n\nElsewhere in the document you say 'upload-pack' or 'receive-pack'.\nI think you should be saying 'git-upload-pack' or 'git-receive-pack'\neverywhere, as these are the formal names in the protocol.\n\nSection 5.2, Capabilities:\n\n> Client sends space separated list of capabilities it wants. It\n> SHOULD send a subset of server capabilities, i.e do not send\n> capabilities served does not advertise. The client SHOULD NOT ask\n> for capabilities the server did not say it supports.\n\nI thought we had said it was client MUST send a subset of server\ncapabilities; client MUST NOT ask for capabilities server did\nnot advertise support of.\n\n> Server MUST ignore capabilities it does not understand. Server MUST\n> NOT ignore capabilities that client requested and server advertised.\n\nI think that's just lazy coding on the server part. If the server\ngets a capability request it can't honor, it MUST abort, it might\ncorrupt the stream to the client.\n\n> 5.2.1. multi-ack\n>\n> The 'multi-ack' capability allows the server to return \"ACK $SHA1\n\nmulti_ack\n\n> Without multi_ack, a client sends have lines in --date-order until\n> the server has found a common base. That means the client will send\n\nExplain --date-order, don't assume the reader knows it.\n\nI'm giving up for now. :-)\n\n-- \nShawn.\n"},{"id":"115893","messageId":"200906091139.58509.jnareb@gmail.com","threadId":"19323","inReplyTo":"20090607200617.GA16497@spearce.org","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-09T09:39:56Z","receivedAt":"2009-06-09T09:39:56Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 7 June 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n\n> > > ### Fetching Data with Upload Pack ###\n> ...\n> > Although fetching via SSH protocol is, I guess, much more rare than\n> > fetching via anonymous unauthenticated git:// protocol,\n> \n> Actually, fetching via SSH might be quite common, think about all of\n> those companies using Git internally... they are running something\n> like Gitosis or Gerrit Code Review, both of which support SSH only\n> access to the hosted repositories.\n\nI am blind... I forgot that Git is not only used for F/OSS software,\nbut also for developing proprietary code in company intranets; here\nyou have limited number of people you want to have read access to\nrepository.\n\n> \n> > it _might_ be\n> > good idea to tell there that fetching via SSH differs from above\n> > sequence that instead of opening TCP connection to port 9418 and\n> > sending above packet, and later reading from and writing to socket,\n> > \"git clone ssh://myserver.com/srv/git/project.git\" calls\n> > \n> > \tssh myserver.com git-upload-pack /srv/git/project.git\n> > \n> > and later reads from standard output of the above command, and writes\n> > to standard input of above command.\n> \n> Yes, this should be mentioned. We actually should document in\n> the protocol specifiction how we fork SSH, and what the SSH server\n> should then be presenting as the command line.\n> \n> I've run into problems with hosting sites like GitHub and Gitoriuous\n> not correctly honoring some ssh invokes, because they use the forced\n> command execution model and were handling only one case that could\n> be presented to them.\n\nCan you offer some details? Or is it out of scope of git pack protocol\ndescription, and more about correctly implementing SSH protocol and\nremote command invocation in it?\n\n> \n> > The rest of exchange is _identical_ for git:// and for ssh:// (and\n> > I guess also for file:// pseudoprotocol).\n> \n> Yes, the file:// pseduoprotocol works by forking a child to run the\n> `git-upload-pack /srv/git/project.git` executable and runs a pair\n> of pipes between them, just like ssh:// does when it spawns off\n> the ssh client process.\n\nThat would be nice information to have for people (re)implementing Git,\nI think.\n\nSidenote: it will be the same for planned \"smart\" HTTP protocol, but \nfor the fact that HTTP is stateless, and additionally some kind of state\ninformation would have to be passed.\n\n> > Footnote: [pkt-line format] somewhat reminds / resembles 'chunked' transfer\n> > encoding used in HTTP[1], although there are differences.\n> > http://en.wikipedia.org/wiki/Chunked_transfer_encoding\n> \n> This is not worth mentioning. pkt-line is different enough that\n> it may just confuse the reader.\n\nO.K. \n\nI mentioned it because it also uses hexadecimal for length.\n\n> \n> > Below there is (for completeness) list of git-upload-pack\n> > capabilities, with short description of each:\n> > \n> > * multi_ack (for historical reasons not multi-ack)\n> ...\n> > See the thread for more details (posts by Shawn O. Pearce and by\n> > Junio C Hamano).\n> \n> This really needs a diagram example, like the one I drew, to\n> explain the concept. Its really hard to grasp from just reading\n> that paragraph what that implies, especially if you are implementing\n> a client or a server.\n\nWhile I don't think that one would have to describe Git object model,\nand Git repository storage model (the Git repository storage model, \ni.e. loose object format, and packfile and packfile index format,\nand everything else in .git should be described in separate RFC-like\ndocument, in my opinion), it would be helpful to describe \"history DAG\"\nmodel Git uses, and a bit about revision walking. What use would be\ndescribing git pack protocol, if the idea behind it, namely coming up\nwith optimal packfile to send won't be understood?\n\n> > * no-progress\n> > \n> > Client should use it if it was started with \"git clone -q\" or\n> > something, and doesn't want that side brand 2. We still want\n> \n> typo, should be \"... side band 2.\" :-)\n> \n> > sideband 1 with actual data (packfile), and sideband 3 with error\n> > messages.\n> \n> Also, this capability really only makes sense if side-band or\n> side-band-64k was requested. IOW, a sane client wouldn't ask\n> for this if it doesn't support side-band.\n\nRight. \"no-progress\" makes sense only in context of sideband, currently\n\"side-band\" and \"side-band-64k\". For server it means that it MUST send\n(currently) only streams 1 (data) and 3 (fatal error); conversely it\nMUST NOT send stream 2 (progress).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"115907","messageId":"20090609142852.GL16497@spearce.org","threadId":"19323","inReplyTo":"200906091139.58509.jnareb@gmail.com","subject":"Re: Comments pack protocol description in \"Git Community Book\" (second round)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-09T14:28:52Z","receivedAt":"2009-06-09T14:28:52Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> On Sun, 7 June 2009, Shawn O. Pearce wrote:\n> > \n> > I've run into problems with hosting sites like GitHub and Gitoriuous\n> > not correctly honoring some ssh invokes, because they use the forced\n> > command execution model and were handling only one case that could\n> > be presented to them.\n> \n> Can you offer some details? Or is it out of scope of git pack protocol\n> description, and more about correctly implementing SSH protocol and\n> remote command invocation in it?\n\nFor URI user@site:project.git the following should all succeed:\n\n 1) ssh user@site \"git-receive-pack project.git\"\n 2) ssh user@site \"git receive-pack project.git\"\n\n 3) ssh user@site \"git-receive-pack 'project.git'\"\n 4) ssh user@site \"git receive-pack 'project.git'\"\n\nNote that the command name can be dash or dashless, and the project\nname can be wrapped in single quotes, or not wrapped in single\nquotes. C Git tends to create form 3 by default. JGit also tries\nto use form 3 by default, but I've heard some reports from users\nsaying it produced one of the other forms.\n\nFor ssh://user@site/project.git the following should succeed:\n \n 1) ssh user@site \"git-receive-pack /project.git\"\n 2) ssh user@site \"git receive-pack /project.git\"\n\n 3) ssh user@site \"git-receive-pack '/project.git'\"\n 4) ssh user@site \"git receive-pack '/project.git'\"\n\nIf you are a site like GitHub, where '/' means nothing to you\nbecause all repositories are scoped by user, the extra leading '/'\nappears here simply because of the style of URI used, and the '/'\nshould be dropped before evaluating the path.\n\nGitHub had a problem with ssh:// URIs, its fixed now. I think\nGitorious had a problem with the command name, but that's fuzzy\nin my memory.\n\n-- \nShawn.\n"},{"id":"116220","messageId":"200906131130.57709.jnareb@gmail.com","threadId":"19323","inReplyTo":"d411cc4a0906061458g494d80dbwe3a5358edfd1d49e@mail.gmail.com","subject":"Re: Comments pack protocol description in \"RFC for the Git Packfile Protocol\" (long)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-06-13T09:30:55Z","receivedAt":"2009-06-13T09:30:55Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 6 June 2009, Scott Chacon wrote:\n\n> In addition to that, I started taking a shot at putting together an\n> RFC formatted documentation of this protocol as was requested. I may\n> have _way_ missed the mark on what you were looking for originally,\n> it's hard to say, not having read a lot of RFC documents - I probably\n> ended up writing in a more bookish format rather than a technical\n> spec, but whatever - maybe you'll find it helpful or can fix it up to\n> more what you were expecting. I'm not done with it - some of it is\n> still basically unformatted comments from this previous thread, but at\n> least it's laid out roughly how I thought it might be useful and I\n> have fleshed out a lot of it. You can find the RFC text output\n> document here:\n> \n> http://git-scm.com/gitserver.txt\n> \n> And the xml doc I generated it from here:\n> \n> http://github.com/schacon/gitserver-rfc\n\nIt would be nice to have RFC text output document alongside XML source\nin gitserver-rfc document, e.g. following example of 'man' and 'html'\nbranches in git.git repository; which means having unrelated 'txt'\nbranch with text version of document for review. Not everybody has\nand wants to install tools required to turn XML into pretty RFC-like\ntext document... and it is hard to read XML.\n\nAs it is now, I find it much easier to read plain text formatted\noutput, and in current situation I cannot be sure that I am working on\n(where working on means in this case commenting) the most recent\nversion of this RFC draft. See also comment about embedding version\nnumber in plain text version.\n\n> \n> Perhaps if we're going to spend time getting this all correct, we\n> should get a standalone technical doc all agreed upon, then I can\n> relatively easily extract what's needed into that chapter of the\n> Community book.\n> \n> Thoughts?\n> \n> Scott\n\nHere are my thoughts and comments about the RFC. Below the quoted\ntext is gitserver.txt from around 2009-06-10, not Scott Chacon email.\n\n\nWell, I think having _detailed_ technical documentation of git pack\nprotocol would help tremendously implementers (whether it is\nreimplementation of git, or just a git-server equivalent). What exact\nformat it would be is not that important, I think...\n\n------------------------------------------------------------------------\n> Internet Engineering Task Force S. Chacon\n> Internet-Draft GitHub\n> Intended status: Informational June 6, 2009\n> Expires: December 8, 2009\n\nWhy is expire date set? Is it required? Why this date, by the way?\nIs it some mandated/consensus expiry length, or is it date you plan\nto revise this draft?\n\nBy the way, it would be nice if you tagged the source (e.g. v0.1), and\nin during conversion from XML to RFC in text format embed version\nnumber (GIT-VERSION-GEN) somewhere... perhaps in pseudo-filename.\n\n>\n> Git Server Protocol\n> git-server-protocol-01\n\nAbout naming: RFC drafts found at IETF have draft-- prefix in\na draft \"filename\", e.g. \n draft-mirashi-url-irc-01.txt, \n draft-ietf-atompub-format-11.txt, \n draft-templin-autoconf-dhcp-38.txt,\n draft-rfc-editor-rfc2223bis-08.txt,\n(I am not sure if the .txt extension should be stated or not).\n\nFor example if the root commit of gitserver-rfc was tagged with v0.1,\nthen version which is 2 commits later could have pseudo filename look\nlike this: 'draft-schacon-git-server-protocol-01_2_ge036f1.txt', or\nsomething like that (but the exact version might be put in other\nplace, and the filename might be then 'draft-*-01.txt').\n\n\nShouldn't it be \"Git Pack Protocol\" (or \"Git Pack Protocol Exchange\",\nor \"Git Packfile Protocol\" as you described gitserver-rfc repository)\nrather than \"Git Server Protocol\": the protocol is the same for\nfile:// URLs (over pipe), ssh:// URLs (over SSH), and git:// (over\nsocket using git-daemon)? Although I guess the naming (the name is\nthe hardest thing... ;-) of this RFC draft could be left for later...\n\n\n> Status of this Memo\n>\n> By submitting this Internet-Draft, each author represents that any\n> applicable patent or other IPR claims of which he or she is aware\n> have been or will be disclosed, and any of which he or she becomes\n> aware will be disclosed, in accordance with Section 6 of BCP 79.\n[...]\n\nThis boilerplate is fairly short, and I guess it could remain; but\nwouldn't it be a good idea (if it is possible) to leave adding larger\nparts of boilerplate text, which is required for RFC, but is not\nnecessary for draft?\n\n\n> Abstract\n>\n> This documents the Git version control system packfile based server\n> protocol. It describes expected behaviour of client and server and\n> best current practices to help avoid pitfalls when implementing Git\n> daemon or SSH based servers in other language implementations. It\n> will describe the data structures underlying Git repositories, how\n> that data is compressed into a packfile and how the contents of that\n> packfile are negotiated and transferred. This does not cover the\n> HTTP based Git server protocols.\n\nGood abstract. \"Packfile based server protocol\" is a good name. \nI think however that the goal of git server/pack protocol exchange\nshould be stated explicitly: it is coming up with optimal packfile\nto send.\n\nI am not sure if description of Git repositories data structures (and\nlayout) wouldn't be better left for separate RFC (or refer to existing\ndocumentation of those). What we need for git server protocol (git\npack protocol) is actually a subset of this information, I think.\n\n\"This does not cover the HTTP based Git server protocols.\" is not\nentirely correct. What is not covered are \"commit walker\" based\nprotocols, like for example HTTP, but also FTP; also other \"dumb\"\nprotocols like (deprecated) rsync. Those \"commit walker\" and other\n\"dumb\" protocols do not require Git aware server (but they do require\nextra helper info), so they are not strictly \"Git _server_\nprotocols\". (That is a bit splitting hairs.)\n\n[Table of contents]\n\n> 1. Introduction\n>\n> The Git SCM is a snapshot based distributed version control system.\n> Each clone of each repository can synchronize with other nodes if\n> they have read or write access to them. The two most common\n> protocols that these communications happen over are the custom 'Git'\n> protocol and over SSH. In both of these cases, the communication\n> happens between the 'send-pack' process on the client side and\n> 'receieve-pack' process on the server in the case of pushing changes\n> from the client to the server. For fetching changes from the server\n> to the client, the 'fetch-pack' process on the client communicates\n> with an 'upload-pack' process on the server. This document will\n> describe the ways in which these pairs of processes communicate.\n\nWhat I find lacking is reference to RFC 2119 (which covers meaning of\nMUST, etc.) and RFC 5234 (which covers ABNF). It was present in XML\nsource, but was subsequently removed.\n\nTAP (Test Anything Protocol) draft uses the following wording:\nhttp://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard\n\n Conventions Used In This Document\n =================================\n\n The key words \"MUST\", \"MUST NOT\", \"REQUIRED\", \"SHALL\", \"SHALL NOT\",\n \"SHOULD\", \"SHOULD NOT\", \"RECOMMENDED\", \"MAY\", and \"OPTIONAL\" in this\n document are to be interpreted as described in [RFC2119].\n\n The grammatical rules in this document are following ABNF and are to\n be interpreted as described in [STD68].\n\nThe wording for RFC 2119 reference is given in this RFC. For\nreference to ABNF standard one has to come up with ones own wording.\nAnother example would be:\n\n All the mechanisms specified in this document are described in both\n prose and an augmented Backus-Naur form (ABNF). It is described in\n detail in [RFC5234].\n\nRFC 2616 (Hypertext Transfer Protocol -- HTTP/1.1) has section titled\n\"1.2 Requirements\" with reference to RFC 2119, and the following\nparagraph:\n\n An implementation is not compliant if it fails to satisfy one or more\n of the MUST or REQUIRED level requirements for the protocols it\n implements. An implementation that satisfies all the MUST or REQUIRED\n level and all the SHOULD level requirements for its protocols is said\n to be \"unconditionally compliant\"; one that satisfies all the MUST\n level requirements but not all the SHOULD level requirements for its\n protocols is said to be \"conditionally compliant.\"\n\n\n\nTAP standard draft has there \"Conventions Used In This Document\"\nsubsection, with additional reference to ABNF, as stated above. Other\nRFCs (like RFC 3501, Internet Message Access Protocol - Version 4rev1) \nhave it specified/referenced in the section where it is used for the\nfirst time, e.g. (from RFC 3501):\n\n 9. Formal Syntax\n ================\n\n The following syntax specification uses the Augmented Backus-Naur\n Form (ABNF) notation as specified in [ABNF].\n\n\nWhat is also missing is section titled \"Terminology\" or \"Definitions\",\nwhich can also be put in \"Conventions Used in This Document\" section,\nwhere we would have to define _at least_ (I think) 'client', 'server',\n'connection', 'request', probably 'channel' or 'sideband', 'capability'.\n\nAlso, if we are using \"C: \"/\"S: \" prefix convention, we should state\nit explicitly, e.g. like it is written in RFC 3501 (IMAP):\n\n In examples, \"C:\" and \"S:\" indicate lines sent by the client and\n server respectively.\n\nor like in draft RFC3920bis (XMPP core):\n\n In examples, lines have been wrapped for improved readability,\n \"[...]\" means elision, and the following prepended strings are used\n (these prepended strings are not to be sent over the wire):\n\n o C: = a client\n o S: = a server\n\n\nAfter list of conventions there should be _overview_ of protocol,\nbefore details; RFC 2616 (HTTP/1.1) has there \"Overall Operation\",\nRFC 3501 (IMAP v4) has \"Protocol Overview\", and XMPP draft has\n\"Overview\" section inside \"Architecture\" chapter.\n\n\n> 2. Git Data\n>\n> Git has a relatively simple data format for storing it's objects.\n> There are four different types of objects that Git stores and these\n> make up nearly all the data that is transferred between a Git client\n> and server. The four object types are the 'blob', the 'tree', the\n> 'commit' and the 'tag'. [...]\n\nOn one hand side I think that definitions of Git object model and Git\nrepository structure, file formats and layout should be either put in\n_separate_ RFC, or (for the time being at least) simply refer to\nexisting documentation (it is only pack protocol which was lacking\nmuch in matter of detailed description).\n\nOn the other hand side some of this information is actually required\nto understand git server/pack protocol. One needs to know about\nreachability (of commits) to understand git pack protocol exchange,\nand a way to come up with minimal set of commits; also one would have\nto know how to interpret ASCII-art diagrams we use to visualize\nhistory (commits parentage). One needs to know about trees and blobs\n(but not necessarily details about their exact format) and about Git\nassumptions to understand how one should arrive at minimal set of\nobjects (in ordinary case, and in 'shallow' case). One needs to\nunderstand what tags are (but not necessarily details about their\nformat) to know e.g. what \"include-tag\" capability is about. One\nneeds to understand what packfile is about to know what is thin pack,\nand what \"ofs-delta\" capability is about.\n\n[...]\n\n> 2.1. The SHA-1 ID\n>\n> The Git database operates as a key-value store, where each object\n> that is put into the database is given an ID and then can be\n> retrieved from the database by that ID. The ID is calculated as the\n> SHA-1 checksum of the content being stored plus a small header\n> appended to it of the format: [...]\n\nThe fact that Git repository is content-addressed object database,\nusing SHA-1 identifiers is fairly important; for pack protocol\ndescription what matters is SHA-1 ids of commits and of tags, only.\n\nBTW. it is prepended, and not appended.\n\n[...]\n\n> 2.2.5. Git Object Model\n>\n> The Git object model then are tags that point to commits, \n\nAnd that misunderstanding led to GitHub not supporting tags to blobs\n(like e.g. junio-gpg-pub tag in git repository) :-). Tag (both tag\nreference, and tag object) can point to _any_ kind of object, be it\ncommit (most common), tag (chain of trust, retagging etc.), blob (like\ne.g. public GPG keys) or trees (in Linux kernel first version didn't\nhave commit, so tree was tagged).\n\n> which point to zero or more commits and a single tree, which\n> points to one or more trees and/or blobs.\n\nSidenote: since submodules trees can point to commits; what is\nimportant for pack-protocol considerations is that those commits do\nnot enter reachability considerations.\n\n>\n> +---+ +--+\n> v | v |\n> +-----+ +--------+ +--------+ +--------+\n> | Tag | --> | Commit | --> | Tree | --> | Blob |\n> +-----+ +--------+ +--------+ +--------+\n>\n> The Git Object Model\n>\n> This creates a directed acyclic graph that can represent the project\n> state at any point.\n\nBecause tags can point at any kind of objects (though this is rare)\nabove diagram is not entirely true. Also above diagram at least for\nme seems to imply that there are loops in DAG of objects, while loops\nare impossible (well, unless one screws up with grafts...) in Git.\n\nBut it would be hard to come up with ASCII-art diagram of DAG of\nobjects in Git repository; perhaps we can use ABNF to describe it (it\nwould be used to describe how objects refer to one another, not to\ndescribe some syntax).\n\n[...]\n\nBut I do not think that describing details of object format (loose,\npacked, object) is necessarily here. I'd rather leave it to existing\ndocumentation (which describes Git model, and loose format, and pack\nformat with its indexes quite well), or separate RFC.\n\n\n> 2.2.5.1. The Commit Graph\n>\n> Importantly for calculating data needs later on, the commit objects\n> by themselves are also a directed acyclic graph. [...]\n\nThis is very important section, and I think necessary to understand\nwhat git pack protocol is about. We also introduce here ASCII-art\ndiagram convention for representing history in examples and\nexplanations. \n\n\n> 2.3. Git References\n>\n> The last major concept in the Git data structure is the reference. A\n> reference is like a tag that moves. When users work on a branch in\n> Git, the branch reference that is currently checked out is moved\n> forward to point to each new commit that is created. So in Git, a\n> branch is really just a pointer to the latest commit on that branch -\n> the rest of the commits are obtained by walking the SHA-1 values one\n> commit at a time.\n>\n> +-- E <= topic1\n> /\n> A -- B -- C -- G <= master\n> \\\n> +-- D -- F <= topic2\n>\n> Commit Graph with References\n\nI don't think that it is good description. What's more such level of\ndetail is not necessary; what we have to know is that references are\nlocal symbolic names for objects, usually for commits in DAG of\nrevisions. Perhaps introduction of HEAD, refs/heads/* (which must\npoint to commits) and refs/tags/* would also be needed.\n\n[...]\n\n> 3. Git Packfile Format\n>\n> Once the client and the server figure out what objects need to be\n> transferred from one system to another, it will put all of those\n> objects into a \"packfile\". This packfile is then streamed from one\n> system to the other.\n>\n> The packfile itself is a very simple format. There is a header, a\n> series of packed objects (each with it's own header and body) and\n> then a checksum trailer. The first four bytes is the string 'PACK',\n> [...]\n\nI am not sure if detailed description of packfile format is really\nnecessary. We can always refer to Documentation/technical/pack-format.txt\nand Documentation/technical/pack-heuristics.txt\n\nBy the way, if we are to describe details of packfile format, perhaps\nwe should use format described in section \"5.2. Protocol Data\nDefinitions\" of \"RFC Style Guide\"[1], based on RFC 791, isn't it?\n\n[1]: http://www.rfc-editor.org/rfc-style-guide/rfc-style-manual-08.txt\n\n\n> 3.1. Deltified Objects\n>\n> There are two object types that are new here - the delta object\n> types. These are object data that are deltas of existing objects,\n> saving space in the storage. The instance that creates the packfile\n> determines which objects it wants to deltify, if any, in order to\n> save space. It is possible to send packfiles with no delta objects\n> in it, though it often saves quite a bit of space.\n>\n> For the two delta object representations, the data portion contains\n> something that identifies which base object this delta representation\n> depends on, and then the delta to apply on the base object to\n> resurrect this object.\n>\n> REF_DELTA uses 20-byte hash of the base object at the beginning of\n> data, while OFS_DELTA stores an offset within the same packfile to\n> identify the base object. In either case, two important constraints\n> a reimplementor must adhere to are:\n> [...]\n\nREF_DELTA vs OFS_DELTA is required to understand \"ofs-delta\"\ncapability; delta object and delta base is required to understand thin\npacks. But do we need more?\n\n.......................................................................\nThe majority of comments touches the following chapter. I haven't\nexamined previous chapters in more detail.\n\n> 4. Protocols\n>\n> There are two transports over which the packfile protocol is\n> initiated. The Git protocol is a simple, unauthenticated server that\n> simply takes the command (almost always 'upload-pack', though Git\n> servers can be configured to be globally writable, in which 'receive-\n> pack' initiation is also allowed) with which the client wishes to\n> communicate and executes it and connects it to the requesting\n> process. The other transport is the SSH protocol, in which the\n> client basically just runs the 'upload-pack' or 'receive-pack'\n> process over the SSH protocol.\n\nSidenote: Actually there are three transports over which packfile\nprotocol is initiated. There are: \n1.) transport over TCP socket, with git-daemon server being thin\n wrapper around whitelist of allowed commands, which uses\n git://git.example.com/repo.git URLs,\n2.) transport over SSH, where clients run 'upload-pack' or\n 'receive-pack' process over SSH protocol,\n3.) transport over pipe, where client run 'upload-pack' or\n 'receive-pack' process locally, file://git.example.com/repo.git \n (pseudo)protocol URL.\nWe do not want to close possibility of other transports, like JGit's\namazon-s3:// protocol.\n\nSidenote: The set of commands which can be run via git protocol (via\nTCP socket) from git-daemon is not limited to 'upload-pack';\n'upload-archive' can be enables as well.\n\n\n> 4.1. Packet Line Format\n>\n> Some data transmission in Git is done in what is called 'packet-line'\n> format. \n\nShould we mention that it is called pkt-line or pkt_line in sources\nand in other documentation?\n\n> This is where each line of data sent is prepended with the\n> four byte hex encoded length of the rest of the payload being sent.\n ^^^^^^^\n \\--- this is wrong!\n\nThe four hexadecimal characters length of packet line is length of the\nwhole line, including length prefix, and not only of payload.\n\n> This way the side receiving data can read 4 bytes and then know how\n> much more data is coming in that request.\n\nTherefore it is \"how much data\", not \"how much more data\".\n\n>\n> pkt-length = 4HEXDIGIT ; length of pkt-payload\n> pkt-line = pkt-length pkt-payload [ LF / CR ]\n\nThis is wrong. I would use the following top-down definition:\n\n pkt-line = pkt-length pkt-payload [ LF / CR / NUL ]\n pkt-length = 4HEXDIG ; length of pkt-line\n\nwhere\n\n NUL = %x00 ; \\0\n\nPerhaps even\n\n pkt-payload = *OCTET ; data\n\nOCTET, HEXDIG and LF are defined in \"Core rules\" (Appendix B) of ABNF\nstandard. I think that _any_ terminator is allowed, but we should use\nonly LF (CR and NUL is used in some special cases); I am not convinced\nif we should state that there is/can be terminator explicitly.\n\nThis definition does not cover special case (which should be\ndescribed) of\n\n pkt-flush = \"0000\"\n\n>\n> In some cases Git will use a sideband packet-line format, where each\n> line is transmitted with the hex length prepended, followed by the\n> sideband channel (one byte) that the data is meant for, followed by\n> the actual data.\n>\n> pkt-length = 4HEXDIGIT ; length of pkt-sb-payload\n> sideband-ch = %d01-%d03\n> pkt-line-sb = pkt-length sideband-ch pkt-payload [LF/CR]\n\nThis also contains some errors.\n\n pkt-line-sb = pkt-length sideband-ch pkt-payload [LF/CR]\n pkt-length = 4HEXDIGIT ; length of pkt-line-sb\n sideband-ch = %d01-%d03\n\nI don't think that there is situation where we use NUL for terminator\nfor sideband packet-line format. Also the CR terminator appear only\nin sideband packer-line format for 2 channel, \"progress message\",\nisn't it?\n\n>\n> When a sideband is used, 2 means \"progress messages, most likely\n> suitable for stderr\". 1 means \"pack data\". 3 means \"fatal error\n> message, and we're dead now\". No other channels are used or valid.\n>\n> For the hex encoding, client and server SHOULD use lowercase, but\n> MUST accept mixed case (do case insensitive parsing of hex4).\n\nHere, as you can see, we make use of RFC 2119, so we should reference\nit somewhere at the beginning of RFC, as I have stated way above.\n\n>\n> 4.2. Git Protocol\n>\n> The Git protocol starts off by sending \"git-receive-pack 'repo.git'\"\n> on the wire using the pkt-line format, followed by a null byte and a\n> hostname parameter, terminated by a null byte.\n>\n> 0032git-upload-pack /project.git\\0host=myserver.com\\0\n\nYou didn't mention that above is only an example. You didn't provide\nwith [example] invocation that results with such request. Also it\nwould be better to use 'example.com' or 'git.example.com' as hostname,\nsee RFC 2606 (\"Reserved Top-Level DNS Names\" [TLD99]), to avoid\naccidental conflicts.\n\nWe can use ABNF to encode it\n\n git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]\n request-command = 'git-upload-pack' / 'git-receive-pack' /\n 'git-upload-archive' ; case sensitive\n pathname = *( %x01-ff ) ; exclude NUL\n host-parameter = 'host' \"=\" hostname [ \":\" port ] \n\nOne should probably take a look at RFC for URLs/URIs for definitions\npertaining to host, pathname, URL, absolute and relative, etc.\n\nHere I used for simplicity extension to ABNF that all characters\ninside single quotes, like for example 'host' above are case\nsensitive, as opposed to ABNF strings like \"A\" or \"cmd\" which are case\ninsensitive (in US-ASCII).\n\nOtherwise one would have to specify 'host' as e.g. %x68.6f.73.74\nor %d104.111.115.116, and similar for other cases.\n\n>\n> Currently only 'host' is supported in the extra information. It's\n> for the git-daemon name based virtual hosting. See --interpolated-\n> path option to git daemon, with the %H/%CH format characters.\n\nActually from the discussion on git mailing list (which was after you\nwrote the above) 'host' information is the only ALLOWED, not only the\nonly supported extra information (hence my definition).\n\n>\n> Basically what the Git client is doing to connect to an 'upload-pack'\n> process on the server side over the Git protocol is this:\n>\n> $ echo -e -n \\\n> \"0039git-upload-pack /schacon/gitbook.git\\0host=github.com\\0\" |\n> nc -v github.com 9418\n\nNice to have example here, but in RFC you should, I think, not use\nreal-life URLs, as per \"Instructions to RFC Authors\" (The use of URLs\nin RFCs is discouraged, because many URLs are not stable references.)\n\n>\n> 4.3. SSH Protocol\n>\n> Initiating the upload-pack or receive-pack processes over SSH is\n> simply executing the binary on the server via SSH remote execution.\n> It is basically equivalent to running this:\n>\n> $ ssh git.example.com 'git-upload-pack /project.git'\n>\n\nThis of course depends on the command used, whether it uses URL-like\nrequest, or scp-like / ssh-like request (possibly with relative path).\nBut you write about this below.\n\n> For a server to support Git pushing and pulling for a given user over\n> SSH, that user needs to be able to execute one or both of those\n> commands via the SSH shell that they are provided on login. On some\n> systems, that shell access is limited to only being able to run those\n> two commands, or even just one of them.\n>\n> In an ssh:// format URI, it's absolute in the URI, so the '/' after\n> the host name (or port number) is sent as an argument, which is then\n> read by the remote git-upload-pack exactly as is, so it's effectively\n> an absolute path in the remote filesystem.\n>\n> git clone ssh://user@example.com/project.git\n> |\n> v\n> ssh user@example.com 'git-upload-pack /project.git'\n>\n> In a \"user@host:path\" format URI, its relative to the user's home\n> directory, because the Git client will run:\n>\n> git clone user@example.com:project.git\n> |\n> v\n> ssh user@example.com 'git-upload-pack project.git'\n>\n>\n> 5. Fetching Data From a Server\n>\n> When one Git repository wants to get all the data that a second\n> repository has, the first can 'fetch' from the second. This\n> operation determines what data the server has that the client does\n> not then streams that data down to the client in packfile format.\n\nI know this is only a draft, but this paragraph could have been\nwritten better...\n\n>\n> The server side binary needs to be executable as 'git-upload-pack'\n> for fetching over SSH, since the Git clients will connect to the\n> server and attempt to run that.\n\nWhy do you single out SSH protocol here?\n\n(Yes, I know it is a draft...)\n\n>\n> The basic communication structure looks like this:\n>\n> # Tell the client current branch heads and the last commit on each\n> S: SHA1 refname\n> S: ...\n> S: SHA1 refname\n> S: # flush -- it's your turn\n> # Tell the server what commits we want, and what we have\n> C: want name\n> C: ..\n> C: want name\n> C: have SHA1\n> C: have SHA1\n> C: ...\n> C: # flush -- occasionally ask \"had enough?\"\n> S: NAK # nope, keep sending 'have's\n> C: have SHA1\n> C: ...\n> C: have SHA1\n> S: ACK\n> C: done\n> S: XXXXXXX -- packfile contents.\n\nThis perhaps is too detailed for the \"Overview\" section (to be\nintroduced)... or perhaps it is not.\n\nSidenote: we should standarize on SHA1 or SHA-1 thorough document.\n\n>\n> 5.1. Initial Server Response\n>\n> When the client initially connects, whether over the SSH or Git\n> transports, the server will immediately respond with a listing of\n> each reference it has (all branches and tags) along with the commit\n> SHA that each reference currently points to.\n>\n> $ echo -e -n \\\n> \"0039git-upload-pack /schacon/gitbook.git\\0host=github.com\\0\" |\n> nc -v github.com 9418\n\nSee the comment about using URLs in RFC.\n\n> Connection to github.com 9418 port [tcp/*] succeeded!\n\nThis is message from 'nc' (to be more exact from 'nc -v'), not from\ngit server.\n\n> 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack \\\n> thin-pack side-band side-band-64k ofs-delta shallow no-progress \\\n> include-tag\n\nHere we should point that we line-wrap lines for convenience, and that\nwe mark such situation with \"\\\" as the last character (line\ncontinuation)... which would be not necessary if we used \"C: \"/\"S: \"\nprefix convention and the convention about line-wrapping here (see\nnote about \"Conventions\" chapter way above).\n\nAlso I think it would be better to specify LF terminator, e.g. \nas \"\\n\" explicitly, as it is not present for \"0000\" pkt-flush line.\n\n> 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n> 003f7217a7c7e582c46cec22a130adf4b9d7d950fba0 refs/heads/master\n> 003cb88d2441cac0977faf98efc80305012112238d9d refs/tags/v0.9\n> 003c525128480b96c89e6418b1e40909bf6c5b2d580f refs/tags/v1.0\n> 003fe92df48743b7bc7d26bcaabfddde0a1e20cae47c refs/tags/v1.0^{}\n> 0000\n\nShould we explain what \"refs/tags/v1.0^{}\" means, or should this be\nleft for later (or earlier, perhaps in separate subsection)?\n\n>\n> Each line is terminated by a \"\\n\" by convention only, which is\n> included in the 4 byte length declaration. If a newline does not\n> terminate the line, the client should not complain.\n\nI think it should read: server SHOULD terminate each non-flush line\nusing LF (\"\\n\") terminator; client MUST NOT complain if there is no\nterminator (is it?).\n\n>\n> The exception is the flush line. A length of \"0000\" means its a\n> flush packet, which has no data payload. An \"\\n\" after the \"0000\"\n> would break the protocol as the server would read that \"\\n\" in a\n> context where it is expecting another pkt-line length declaration.\n> \"\\n\" is not a hex digit, so \"0000\\n\" is horribly horribly broken.\n\nI know it is copy'n'paste from existing discussion (existing post),\nbut something like above shouldn't take place in RFC.\n\n>\n> HEAD is not included if its detached - that is, if HEAD is not a\n> symbolic reference, a pointer to another branch, it is not included\n> in the initial server response.\n\nHere we mix levels of abstraction a bit: a protocol, and a set (well,\na sequence) of references (reference info) returned.\n\nBy the way why detached HEAD is not present? Server do not know\nanything about what refspec client uses. (That is not a comment to\nScott, but to the mailing list in general).\n \n> The client pattern matches the\n> advertisements against the fetch refspec, which is \"refs/heads/\n> *:refs/remotes/origin/*\" by default. HEAD doesn't match the LHS, so\n> it doesn't get wanted by the client.\n\nIn this section we should describe both format of response, and also\n(perhaps below \"Capabilities\" section) ordering and format of\nreference info returned in server response.\n\nFormat of response could be presented in the following way:\n\n refs-response = refs-line [ NUL capabilities ] [ LF ]\n *(refs-line [ LF ] )\n pkt-flush\n refs-line = pkt-length sha1-str SP refname\n sha1-str = 40*HEXDIG\n capabilities = capability *(SP capability)\n\nServer and client SHOULD use lowercase for SHA1, both MUST treat SHA1\nas case-insensitive. (But I think you are writing about this later.)\n\n\n>\n> 5.2. Capabilities\n\nSidenote: IMAP has a list of capabilities (usually defined in separate\n\"upgrade\" RFC) maintained by IANA[3]. Perhaps Git also should have\nsuch list of capabilities in the event some new capabilities will get\ninvented (e.g. symbolic references transfer, or multipack transport)?\n\n[3]: http://www.iana.org/assignments/imap4-capabilities\n\n>\n> On the very first line of the initial server response, the first\n> reference is followed by a null byte and then a list of space\n> delimited server capabilities. These allow the server to declare\n> what it can and cannot do to the client.\n\n\"These allow the server to declare to the client what it can and\ncannot do\" perhaps?\n\n>\n> Client sends space separated list of capabilities it wants. It\n> SHOULD send a subset of server capabilities, i.e do not send\n> capabilities served does not advertise. The client SHOULD NOT ask\n> for capabilities the server did not say it supports.\n>\n> Server MUST ignore capabilities it does not understand. Server MUST\n> NOT ignore capabilities that client requested and server advertised.\n\nThere was discussion about exact meaning of capabilities, and silently\nignoring unknown capabilities... but if I remember correctly you are\nright here.\n\n>\n> 5.2.1. multi-ack\n\nmulti_ack (historical reason).\n\n>\n> The 'multi-ack' capability allows the server to return \"ACK $SHA1\n> continue\" as soon as it finds a commit that it can use as a common\n> base, between the client's wants and the client's have set.\n[...]\n\nGood explanation of this capability will require, I think, much\nexplanation and many ASCII-art diagrams. Perhaps description of\ncapabilities should be moved to appendix?\n\n> 5.2.2. thin-pack\n>\n> Server can send thin packs, i.e. packs which do not contain base\n> elements, if those base elements are available on clients side.\n> Client has thin-pack capability when it understand how to \"thicken\"\n> them adding required delta bases making them independent.\n\nClient doesn't \"have\" capability. Client \"requests\" capability.\nBut client MUST NOT (I think) request 'thin-pack' capability if it\ncannot turn thin packs into proper independent packs.\n\n>\n> Of course it doesn't make sense for client to use (request) this\n> capability for git-clone.\n\nIt turned out in further discussion to be not true. Cloning with\n'--reference' option can make use of thin packs. Besides even client\ncan ask for 'thin-pack' option if it understand it even for initial\ngit-clone, as it is simply ignored when there are no common commits\n(no bases and base objects to exclude from pack).\n\n>\n> 5.2.3. side-band, side-band-64k\n>\n> This means that server can send, and client understand multiplexed\n> (muxed) progress reports and error info interleaved with the packfile\n> itself.\n\n[...]\n> The client MUST send only maximum of one of \"side-band\" and \"side-\n> band-64k\". Server MUST favor side-band-64k if client requests both.\n\nOne should check further discussion about this...\n\n[cut fragment for which I have no comments currently]\n\nThe question is: should we list all capabilities with detailed\ndescription as usual in reference documentation like RFC (draft), or\nshould we move detailed description of current set of allowed\ncapabilities to the appendix? (Note also that set of possible\ncapabilities is different for different commands; it is different for\ngit-upload-pack and for git-receive-pack.)\n\n>\n> 5.3. Client Response\n>\n> Once the client has the initial list of references that the server\n> has, as well as the list of capabilities, it will begin telling the\n> server what objects it wants and what objects it has, so the server\n> can make a packfile that only has the objects that the client needs.\n> The client will also send a list of the capabilities it supports out\n> of what the server said it could do.\n>\n\nIn the example below you use \"C: \" prefix convention, and explicitly\nmark end of line character (\"\\n\" = LF) terminating 'packets'. In the\nfinal version we should, I think, standarize on one convention.\n\n> C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d\\0multi_ack \\\n> side-band-64k ofs-delta\\n\n\nActually here client uses simply SPC (' ') to separate 'want' line\ndata from the list of requested capabilities, not NUL (\"\\0\").\n\n> C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\\n\n> C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\\n\n> C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\\n\n> C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\\n\n> C: 0000\n> C: 0009done\\n\n>\n> S: 0008NAK\\n\n> S: 0023\\002Counting objects: 2797, done.\\n\n> [...]\n> S: 2004\\001PACK\\000\\000\\000\\002 [...]\n>\n> It means the server is answering a prior flush from the client, and\n> is saying \"I still can't serve you, keep tell me more have\".\n\nIt is not clear that \"it\" here refers to NAK response from the server.\n\n\nHere we have the following:\n\n client-cmd-set = want-cmd SP sha1-str [ SP capabilities ] [ LF ]\n *(want-cmd SP sha1-str [ LF ] / pkt-flush )\n *(have-cmd SP sha1-str [ LF ] / pkt-flush )\n pkt-flush \n done-cmd\n want-cmd = 'want'\n have-cmd = 'have'\n done-cmd = 'done'\n\nBut it is a bit hard to express the exchange using only ABNF\nformalism. You would have to understand and describe when flush\npacket can be send, when it should be send, and how to write\nand read from socket as to not deadlock (block).\n\nAlso this is not _complete_ list of commands. See upload-pack.c, and\nbuiltin-fetch-pack.c where there is also \"shallow\", \"unshallow\" and\n\"deepen\" commands listed (from shallow clone / fetch). But perhaps we\nshould describe extra commands with the description of capabilities;\nalternatively we can have here a subsection for each capability that\nallows new client commands (\"shallow\", \"unshallow\" and \"deepen\" for\n\"shallow\" capability) and new server responses (\"ACK %s continue\" for\n\"multi_ack\" capability).\n\nIn the full description you would have to describe not only \"NAK\"\nserver response, but also \"ACK %s\" (and \"ACK %s continue\" for\n\"multi_ack\" capability).\n\n>\n> I have thought that after sending \"0000\" flush line client can wait\n> for NAK or ACK server response... but it is not the case. When I\n> tried to read from server after \"0000\" flush and before \"0009done\\n\",\n> my client (or netcat instance) deadlocked (hung) waiting for server\n> response. I either did a mistake in my fake client, or I don't\n> understand git pack protocol correctly. Should client wait for NAK\n> or ACK from server _only_ after sending maximum number of want/have\n> lines (256 if I remember correctly?)? Yes. It means the client will\n> not issue any more \"have\" lines, as it has nothing further in its\n> history, so the server just has to give up and start generating a\n> pack based on what it knows. After the client receives a \"ACK\" or\n> \"NAK\" for the number of outstanding flushes it still has, *after* it\n> has sent \"done\". This also varies based on whether or not multi_ack\n> was enabled. Its ugly. But basically you keep a running counter of\n> each \"flush\" sent, and then you send a \"done\" out, and then you wait\n> until you have the right number of ACK/NAK answers back, and then the\n> stream changes format.\n\nThis certainly requires cleanup, as it is simple dump of fragments of\nconversation on git mailing list.\n\n>\n> > Should commands such as \"have\", \"want\", \"done\" use lower case or >\n> be case insensitive? These MUST be lowercase. > Should status\n> indicators \"ACK\" and \"NAK\" be upper case, These MUST be uppercase.\n> Though \"ACK %s continue\" MUST be mixed case, as I just wrote it. >\n> Should capabilities be case sensitive, and should they be > compared\n> case sensitive or not? No, they are case sensitive.\n\nThis also requires cleanup, even more so.\n\n>\n> One thing that I did not see mentioned in this thread is that the\n> implementation is allowed to buffer non-flush packets and send\n> multiple of them out with a single write(2). In other words,\n> packet_write() could buffer instead of directly calling safe_write(),\n> while packet_flush() must do safe_write() and make sure it drains. -\n> junio That's one reason why in JGit I call the flush packet of \"0000\"\n> end(), and flush() triggers the drain. JGit buffers everything its\n> writing, but only by one standard \"have\" window IIRC. JGit server\n> code triggers a flush() after side-band channel 2 packet ends, but\n> not an end(), because we only want to drain to the network, not\n> inject a bad \"0000\" packet in the stream.\n\nThese implementation details should probably go later, if they are to\nbe in this RFC at all...\n\n>\n> 0023\\\\002Counting objects: 2797, done.\\n\n> 002b\\\\002Compressing objects: 0% (1/1177) \\r\n> 002c\\\\002Compressing objects: 1% (12/1177) \\r\n> 002c\\\\002Compressing objects: 2% (24/1177) \\r\n> 0053\\\\002Compressing objects: 7% (83/1177) \\r \\\n> Compressing objects: 8% (95/1177) \\r\n> 2004\\\\001PACK\\\\000\\\\000\\\\000\\\\002\\\\000\\\\000\\n\\\\355\\\\225\n> \\\\017x\\\\234\\\\235\\\\216K\\n\\\\302\"...\n> 2005\\\\001\\\\360\\\\204{\\\\225\\\\376\\\\330\\\\345]z\\226\\273\"...\n> ...\n> 0037\\\\002Total 2797 (delta 1799), reused 2360 (delta 1529)\\n\"\n\nThe server response should also be described (as far as it can) using\nABNF for syntax. Server responds NAK or ACK if it has enough\ninformation to generate full set of common commits, and responds with\npackfile and sideband data (progress info and fatal errors) to the\n'done' command from client. There should be, I think, list of\npossible error conditions, and also more details about coming up with\nset of common objects, and stop condition9s).\n\nAlso one has to note that sideband data is sent only if \"side-band\" or\n\"side-band-64k\" capability is requested by client (and server supports\nit), and that length of pkt-line for older \"side-band\" is 1000\ncharacters (max length is \"03e8\"). And that when client requests\n\"no-progress\" no sideband 2 info is sent (but git server still use\nchannel 1 to send packfile).\n\n>\n> Buffering. There are two processes running on the server side, git-\n> pack-objects is producing these messages on its stderr, and the pack\n> data on stdout. Both are actually a pipe read by git-upload-pack in\n> a select loop. If pack-objects can write two messages into the pipe\n> buffer before upload-pack is woken to read them out, upload-pack\n> might find two (or more) messages ready to read without blocking.\n> These get bundled into a single packet, because, why not, its easier\n> to code it that way. Its most common on the end like that, where we\n> dump 100%, and then immediately add the \", done\" and start a new\n> progress meter. Its less likely in the middle, where we try to space\n> out the progress updates to around 1 per second, or 1 per percentage\n> push - determines objects in DAG(C) not in DAG(S) and transfers them\n> via packfile\n\nThose are implementation details; some of this info should be there in\nRFC as security precautions (against deadlocking). Such info would be\ninteresting, as e.g. JGit missed one of hidden in C git assumptions\nand could deadlock due to lower buffer size (or something) in Java...\n\n[removed part about pushing]\n\nNote that for pushing 1.) set of possible server capabilities is\ndifferent, 2.) capabilities are presented in another way (with SP and\nnot NUL as separator).\n\n-- \nJakub Narebski\nPoland\n"}]}