git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 0/8] Hiding refs

From
Duy Nguyen <pclouds@gmail.com>
Date
Mar 18, 2014, 14:27 UTC
Message-ID
<20140318142719.GA9393@lanh>
In-Reply-To
<20140318041739.GA7252@sigill.intra.peff.net>
On Tue, Mar 18, 2014 at 12:17:39AM -0400, Jeff King wrote:
Show 28 quoted lines
> On Fri, Mar 14, 2014 at 05:09:45PM -0700, Shawn Pearce wrote:
> 
> > On Fri, Mar 14, 2014 at 4:30 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> > > On Fri, Mar 14, 2014 at 11:45 PM, Shawn Pearce <spearce@spearce.org> wrote:
> > >>
> > >> You missed the SSH case. It doesn't have this slot to hide the data into.
> > >
> > > Right now we run this for ssh case: "ssh <host> git-upload-pack
> > > <repo-path>". New client can do this instead
> > >
> > > ssh <host> git-upload-pack <repo-path> <client capability flags>
> > 
> > Older servers will fail on this command, and the client must reconnect
> > over SSH, which may mean supplying their password/passphrase again.
> > But its remembered that the uploadPack2 didn't work so this can be
> > blacklisted and not retried for a while.
> 
> I wonder if we could use the environment for optional values. E.g., can
> we run:
> 
>   ssh host GIT_CAPABILITIES=... git-upload-pack <repo-path>
> 
> That will not work everywhere, of course. Sites with git-shell will
> fail, as will sites with custom ssh handler (GitHub, for example, and I
> imagine Gerrit sites, if they support ssh). So we'd still need some
> fallback, but it would work out-of-the-box in a reasonable number of
> cases (and it is really not that different than the http case, which is
> just stuffing the values into $QUERY_STRING anyway :) ).

Aggressively gc'ing linux-2.6 takes forever (and it's being timed so I can't really do any heavy lifting), so I outlined what the new protocol would be instead.

Note that at least for upload-pack client capabilities can be advertised twice: the first time at transport connection level, the second time in the first "want", like in v1. I think this will keep the code change down when we have to support both protocols. Moving all capabilities to the first negotiation may touch many places, but that's for now a baseless guess.

The new capability negotiation is also added for push. We didn't pay much attention to it so far.

I thought about "GIT_CAPABILITIES= git-upload-pack ..." (and actually added it in pack-protocol.txt then deleted). The thing is, if you want to new upload-pack, you would need new git-upload-pack at the remote end that must understand "git-upload-pack <repo> <caps>" already. Making it aware about GIT_CAPABILITIES is extra cost for nothing. And we have to update git-shell to support it eventually.

Well, the "must understand" part is not entirely true. If you make git-daemon pass the early capabilities via GIT_CAPABILITIES too, upload-pack does not have to support "<repo> <caps>" syntax. The upside is if old git-upload-pack ignores this GIT_CAPABILITIES, it'll break the protocol (see below) and can print friendly error messages. git-daemon has no way of printing friendly messages because it can't negotiate side-band.

I'm still not sure. But we should support either way, not both. Anyway the text for new protocols:

-- 8< --
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 79e5768..c329eb1 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2084,6 +2084,16 @@ remote.pushdefault::
 	`branch.<name>.remote` for all branches, and is overridden by
 	`branch.<name>.pushremote` for specific branches.
 
+remote.useUploadPack2::
+	Set to "always" to use only upload-pack version 2, "never" to
+	always use the original "upload-pack", "auto" to use the
+	original protocol, but if the remote claims it support version
+	2, then set "remote.<name>.useUploadPack2" to
+	"always". Default to "auto".
+
+remote.<name>.useUploadPack2::
+	Override remote.useUploadPack2 per remote.
+
 remote.<name>.url::
 	The URL of a remote repository.  See linkgit:git-fetch[1] or
 	linkgit:git-push[1].
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index 39c6410..3db4219 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -40,15 +40,22 @@ hostname parameter, terminated by a NUL byte.
 
    0032git-upload-pack /project.git\0host=myserver.com\0
 
+Some service may accept an extra argument (e.g. upload-pack version
+2). The extra argument must follow "host".
+
+   0042git-upload-pack /project.git\0host=myserver.com\0flags=someflags\0
+
 --
-   git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]
+   git-proto-request = request-command SP pathname NUL
+		       [ host-parameter NUL [ flags-parameter NUL ] ]
    request-command   = "git-upload-pack" / "git-receive-pack" /
 		       "git-upload-archive"   ; case sensitive
    pathname          = *( %x01-ff ) ; exclude NUL
    host-parameter    = "host=" hostname [ ":" port ]
+   flags-parameter   = "flags=" *( %x01-ff ) ; exclude NULL
 --
 
-Only host-parameter is allowed in the git-proto-request. Clients
+No other parameters are allowed in the git-proto-request. Clients
 MUST NOT attempt to send additional parameters. It is used for the
 git-daemon name based virtual hosting.  See --interpolated-path
 option to git daemon, with the %H/%CH format characters.
@@ -77,6 +84,11 @@ It is basically equivalent to running this:
 
    $ ssh git.example.com "git-upload-pack '/project.git'"
 
+Some service may accept an extra argument (e.g. upload-pack version
+2). The extra argument is appended, e.g.
+
+   $ ssh git.example.com "git-upload-pack '/project.git' 'extra-flags'"
+
 For a server to support Git pushing and pulling for a given user over
 SSH, that user needs to be able to execute one or both of those
 commands via the SSH shell that they are provided on login.  On some
@@ -124,6 +136,20 @@ has, the first can 'fetch' from the second.  This operation determines
 what data the server has that the client does not then streams that
 data down to the client in packfile format.
 
+Initial capability negotiation
+------------------------------
+
+When the client connects to the server with the extra argument,
+upload-pack version 2 is used. Otherwise the original version is
+used. Unless explicitly stated, the original version is implied.
+
+When the client initially connects to the server using upload-pack
+version 2, the server MUST reply with one pkt-line describing its
+capabilities. Capabilities that are recognized by both ends are
+immediately effective.
+
+By default, upload-pack version 1's reference discovery will follow
+unless some capability makes it different.
 
 Reference Discovery
 -------------------
@@ -447,9 +473,14 @@ Reference Discovery
 
 The reference discovery phase is done nearly the same way as it is in the
 fetching protocol. Each reference obj-id and name on the server is sent
-in packet-line format to the client, followed by a flush-pkt.  The only
-real difference is that the capability listing is different - the only
-possible values are 'report-status', 'delete-refs' and 'ofs-delta'.
+in packet-line format to the client, followed by a flush-pkt. Or with
+receive-pack version 2, a separate pkt-line containing capabilities is
+sent back, then followed by reference discovery unless some capability
+changes it.
+
+The only real difference is that the capability listing is different -
+the only possible values are 'report-status', 'delete-refs' and
+'ofs-delta'.
 
 Reference Update Request and Packfile Transfer
 ----------------------------------------------
diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt
index e174343..a165286 100644
--- a/Documentation/technical/protocol-capabilities.txt
+++ b/Documentation/technical/protocol-capabilities.txt
@@ -250,3 +250,8 @@ allow-tip-sha1-in-want
 If the upload-pack server advertises this capability, fetch-pack may
 send "want" lines with SHA-1s that exist at the server but are not
 advertised by upload-pack.
+
+uploadpack2
+-----------
+
+upload-pack version 2 is supported.
-- 8< --
Previous: Jeff KingNext: Duy Nguyen
Message 42 of 54 in “Hiding refs”
  1. 0/8 Hiding refsJunio C Hamano, Jan 30, 2013
  2. 1/8 upload-pack: share more codeJunio C Hamano, Jan 30, 2013
  3. 2/8 upload-pack: simplify request validationJunio C Hamano, Jan 30, 2013
  4. 3/8 upload/receive-pack: allow hiding ref hierarchiesJunio C Hamano, Jan 30, 2013
  5. Jeff KingFeb 5, 2013
  6. Junio C HamanoFeb 5, 2013
  7. Jeff KingFeb 6, 2013
  8. Junio C HamanoFeb 6, 2013
  9. 4/8 parse_fetch_refspec(): clarify the codeflow a bitJunio C Hamano, Jan 30, 2013
  10. 5/8 fetch: use struct ref to represent refs to be fetchedJunio C Hamano, Jan 30, 2013
  11. 6/8 upload-pack: optionally allow fetching from the tips of hidden refsJunio C Hamano, Jan 30, 2013
  12. 7/8 fetch: fetch objects by their exact SHA-1 object namesJunio C Hamano, Jan 30, 2013
  13. Jeff KingFeb 5, 2013
  14. Jeff KingFeb 5, 2013
  15. Junio C HamanoFeb 5, 2013
  16. 8/8 WIP: receive.allowupdatestohiddenJunio C Hamano, Jan 30, 2013
  17. Michael HaggertyFeb 5, 2013
  18. Jonathan NiederFeb 5, 2013
  19. Michael HaggertyFeb 5, 2013
  20. Junio C HamanoFeb 5, 2013
  21. Duy NguyenFeb 6, 2013
  22. Junio C HamanoFeb 6, 2013
  23. Jonathan NiederFeb 6, 2013
  24. Michael HaggertyFeb 6, 2013
  25. Junio C HamanoFeb 6, 2013
  26. Ævar Arnfjörð BjarmasonFeb 6, 2013
  27. Junio C HamanoFeb 7, 2013
  28. Jeff KingFeb 7, 2013
  29. Ævar Arnfjörð BjarmasonFeb 7, 2013
  30. Junio C HamanoFeb 7, 2013
  31. Duy NguyenFeb 23, 2014
  32. Jeff KingMar 11, 2014
  33. Junio C HamanoMar 11, 2014
  34. Jeff KingMar 11, 2014
  35. Junio C HamanoMar 11, 2014
  36. Jeff KingMar 11, 2014
  37. Duy NguyenMar 14, 2014
  38. Shawn PearceMar 14, 2014
  39. Duy NguyenMar 14, 2014
  40. Shawn PearceMar 15, 2014
  41. Jeff KingMar 18, 2014
  42. Duy NguyenMar 18, 2014
  43. Duy NguyenMar 18, 2014
  44. Duy NguyenMar 15, 2014
  45. Jeff KingMar 18, 2014
  46. Jeff KingFeb 6, 2013
  47. Junio C HamanoFeb 5, 2013
  48. Junio C HamanoFeb 5, 2013
  49. Michael HaggertyFeb 6, 2013
  50. Jonathan NiederFeb 6, 2013
  51. Michael HaggertyFeb 6, 2013
  52. Jed BrownFeb 7, 2013
  53. Junio C HamanoFeb 9, 2013
  54. Jed BrownFeb 10, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.