v3, 8 partsupload-pack: optionally allow fetching from the tips of hidden refs

The third round.

 - Multi-valued variable transfer.hiderefs lists prefixes of ref
   hierarchies to be hidden from the requests coming over the
   network.

 - A configuration optionally allows uploadpack to accept fetch
   requests for an object at the tip of a hidden ref.

Elsewhere, we discussed "delaying ref advertisement" (aka "expand refs"), but it is an orthogonal feature and this "hiding refs completely from advertisement" series does not attempt to address.

Patch #2 (simplify request validation), #4 (clarify the codeflow), and #5 (use struct ref) are new. The are all long overdue clean-ups for these codepaths.

The last patch is an illustration why it wouldn't make sense to optionally allow pushing into hidden refs, and not meant to be part of the series proper.

For those who missed it, earlier rounds are at:

    http://thread.gmane.org/gmane.comp.version-control.git/213951
    http://thread.gmane.org/gmane.comp.version-control.git/214888

Junio C Hamano (8):
  upload-pack: share more code
  upload-pack: simplify request validation
  upload/receive-pack: allow hiding ref hierarchies
  parse_fetch_refspec(): clarify the codeflow a bit
  fetch: use struct ref to represent refs to be fetched
  upload-pack: optionally allow fetching from the tips of hidden refs
  fetch: fetch objects by their exact SHA-1 object names
  WIP: receive.allowupdatestohidden

 Documentation/config.txt |  23 +++++++++++
 builtin/fetch-pack.c     |  40 +++++++++++++++----
 builtin/receive-pack.c   |  31 +++++++++++++++
 cache.h                  |   3 +-
 fetch-pack.c             | 101 ++++++++++++++++++++++++++++++++---------------
 fetch-pack.h             |  11 +++---
 refs.c                   |  41 +++++++++++++++++++
 refs.h                   |   3 ++
 remote.c                 |  41 ++++++++++---------
 remote.h                 |   1 +
 t/t5512-ls-remote.sh     |   9 +++++
 t/t5516-fetch-push.sh    |  82 ++++++++++++++++++++++++++++++++++++++
 transport.c              |   9 +----
 upload-pack.c            |  86 ++++++++++++++++++++++++----------------
 14 files changed, 374 insertions(+), 107 deletions(-)

We mark the objects pointed at our refs with "OUR_REF" flag in two functions (mark_our_ref() and send_ref()), but we can just use the former as a helper for the latter.

Update the way mark_our_ref() prepares in-core object to use lookup_unknown_object() to delay reading the actual object data, just like we did in 435c833 (upload-pack: use peel_ref for ref advertisements, 2012-10-04).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 upload-pack.c | 31 ++++++++++++++-----------------
 1 file changed, 14 insertions(+), 17 deletions(-)

Long time ago, we used to punt on a large (read: asking for more than 256 refs) fetch request and instead sent a full pack, because we couldn't fit many refs on the command line of rev-list we run internally to enumerate the objects to be sent. To fix this, 565ebbf (upload-pack: tighten request validation., 2005-10-24), added a check to count the number of refs in the request and matched with the number of refs we advertised, and changed the invocation of rev-list to pass "--all" to it, still keeping us under the command line argument limit.

However, these days we feed the list of objects requested and the list of objects the other end is known to have via standard input, so there is no longer a valid reason to special case a full clone request. Remove the code associated with "create_full_pack" to simplify the logic.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 upload-pack.c | 28 +++++++++++-----------------
 1 file changed, 11 insertions(+), 17 deletions(-)

Teach upload-pack and receive-pack to omit some refs from their initial advertisements by paying attention to the transfer.hiderefs multi-valued configuration variable. Any ref that is under the hierarchies listed on the value of this variable is excluded from responses to requests made by "ls-remote", "fetch", "clone", "push", etc.

A typical use case may be

	[transfer]
		hiderefs = refs/pull

to hide the refs that are internally used by the hosting site and should not be exposed over the network.

Because these hidden refs do not count as OUR_REF, an attempt to fetch objects at the tip of them will be rejected, and because these refs do not get advertised, "git push :" will not see local branches that have the same name as them as "matching" ones to be sent.

An attempt to update/delete these hidden refs with an explicit refspec, e.g. "git push origin :refs/pull/11/head", is rejected.

This is not a new restriction. To the pusher, it would appear that there is no such ref, so its push request will conclude with "Now that I sent you all the data, it is time for you to update the refs. I saw that the ref did not exist when I started pushing, and I want the result to point at this commit". The receiving end will apply the compare-and-swap rule to this request and rejects the push with "Well, your update request conflicts with somebody else; I see there is such a ref.", which is the right thing to do. Otherwise a push to a hidden ref will always be "the last one wins", which is not a good default.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/config.txt | 11 +++++++++++
 builtin/receive-pack.c   | 24 ++++++++++++++++++++++++
 refs.c                   | 41 +++++++++++++++++++++++++++++++++++++++++
 refs.h                   |  3 +++
 t/t5512-ls-remote.sh     |  9 +++++++++
 t/t5516-fetch-push.sh    | 24 ++++++++++++++++++++++++
 upload-pack.c            | 14 +++++++++++++-
 7 files changed, 125 insertions(+), 1 deletion(-)

On Wed, Jan 30, 2013 at 10:45:37AM -0800, Junio C Hamano wrote:

In the earlier review, I mentioned making this per-service, but I see that is not the case here. Do you have an argument against doing so?

I'm specifically thinking of the way we do refs/pull at GitHub (which we hide only from receive-pack). I know that you think it would be cleaner to hide those, and at some level I agree. But at the same time, the current mechanism has been in place for some time; changing what we present via upload-pack is likely to break people's workflows. And I have not seen complaints about the current system. So unless there is a compelling reason to do so, I'd rather let the fetcher make the decision.

Gerrit's refs/changes may be a different story, if they have a large enough number of them to make upload-pack's ref advertisement overwhelming.

I'm happy to do the per-service patch on top, but I just expected it here, so I'm wondering if you are against having the feature.

-Peff

Jeff King <peff@peff.net> writes:

Perhaps then I misunderstood your intention. By reminding me of the receive-pack side, I thought you were hinting to unify these two into one, which I did. There is no argument against it.

Immediately after I added github to the set of places I push into, which I think is long before you joined GitHub, I noticed that _my_ repository gets contaminated by second rate commits called pull requests, and I may even have complained, but most likely I didn't, as I could easily tell that, even though I know it is _not_ the only way, nor even the best way [*1*], to implement the GitHub's pull request workflow, I perfectly well understood that it would be the most expedient way for GitHub folks to implement this feature.

I think you should take lack of complaints with a huge grain of salt. It does not suggest much.

This is probably a stale count, but platform/frameworks/base part of AOSP has 3200+ refs; the corresponding repository internal to Google has 60k+ refs (this is because there are many in-between states recorded in the internal repository, even though the end result published to the open source repository may be the same) and results in ~4MB advertisement. Which is fairly significant when all you are interested in doing is an "Am I up to date?" poll.

[Footnote]

*1* From the ownership point of view, objects that are only reachable from these refs/pull/* refs do *not* belong to the requestee, until the requestee chooses to accept the changes.

A malicious requestor can fork your repository, add an objectionable blob to it, and throw a pull request at you. GitHub shows that the blob now belongs to your repository, so the requestor turns around and file a DMCA takedown complaint against your repository. A clueful judge would then agree with the complaint after running a "clone --mirror" and seeing the blob in your repository. Oops?

A funny thing is that you cannot "push :refs/pull/1/head" to remove it anymore (I think in the early days, I took them out by doing this a few times, but I may be misremembering), so you cannot make yourself into compliance, even though you are not the offending party. Your repository is held responsible for whatever the rogue requestor added. That is not very nice, is it?

In an ideal world, I would have chosen to create a dedicated fork managed by the hosting company (i.e. GitHub) for your repository whose only purpose is to house these refs/pull/ refs (the hosting site is ultimately who has to respond to DMCA notices anyway, and an arrangement like this makes it clear who is reponsible for what).

The e-mail sent to you to let you know about outstanding pull requests and the web UI could just point at that forked repository, not your own (you also could choose to leave the outging pull requests in the requestor's repository, but that is only OK if you do not worry about (1) a requestor sending a pull request, then updating the branch the pull request talks about later, to trick you with bait-and-switch, or (2) a requestor sending a pull request, thinks he is done with the topic and removes the repository).

On Tue, Feb 05, 2013 at 07:45:01AM -0800, Junio C Hamano wrote:

What I meant was that there should be transfer.hiderefs, and an individual {receive,uploadpack}.hiderefs, similar to the way we have transfer.unpacklimit. That makes the easy case (hiding the refs completely) easy, but leaves the flexibility for more.

Like this:

Jeff King <peff@peff.net> writes:

Yes, as I said, I misunderstood your intention.

Most parts of the cascaded if/else if/... checked an allowable condition but some checked forbidden conditions. This makes adding new allowable conditions unnecessarily inconvenient.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 remote.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

Even though "git fetch" has full infrastructure to parse refspecs to be fetched and match them against the list of refs to come up with the final list of refs to be fetched, the list of refs that are requested to be fetched were internally converted to a plain list of strings at the transport layer and then passed to the underlying fetch-pack driver.

Stop this conversion and instead pass around an array of refs.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 builtin/fetch-pack.c | 40 ++++++++++++++++++++------
 cache.h              |  3 +-
 fetch-pack.c         | 79 +++++++++++++++++++++++++++++++---------------------
 fetch-pack.h         | 11 ++++----
 transport.c          |  9 ++----
 5 files changed, 89 insertions(+), 53 deletions(-)

With uploadpack.allowtipsha1inwant configuration option set, future versions of "git fetch" that allow an exact object name (likely to have been obtained out of band) on the LHS of the fetch refspec can make a request with a "want" line that names an object that may not have been advertised due to transfer.hiderefs configuration.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/config.txt |  8 +++++++-
 upload-pack.c            | 25 +++++++++++++++++++------
 2 files changed, 26 insertions(+), 7 deletions(-)

Teach "git fetch" to accept an exact SHA-1 object name the user may obtain out of band on the LHS of a pathspec, and send it on a "want" message when the server side advertises the allow-tip-sha1-in-want capability.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 fetch-pack.c          | 22 +++++++++++++++++++++-
 remote.c              | 12 +++++++++++-
 remote.h              |  1 +
 t/t5516-fetch-push.sh | 34 ++++++++++++++++++++++++++++++++++
 4 files changed, 67 insertions(+), 2 deletions(-)

On Wed, Jan 30, 2013 at 10:45:41AM -0800, Junio C Hamano wrote:

Hmm. The UI on this is a little less nice than I would have hoped. Right now if you want a ref outside of refs/heads, it's up to you to configure a refspec or do a one-off fetch of the ref:

  git config --add remote.origin.fetch '+refs/pull/*:refs/pull/*'
  git fetch
  git checkout refs/pull/123/head
  ... inspect the contents ...

Without advertisement, we have to learn that refs/pull/123/head exists out of band. We can no longer fetch all of the refs/pull hierarchy preemptively, but we can in theory grab at least that one ref like this:

  git fetch refs/pull/123/head
  git checkout FETCH_HEAD
  ... inspect the contents ...

But that does not work with your patch; instead you have to learn not just the existence of the ref, but also its sha1. This may seem like a little thing, since you are already learning of the ref out-of-band, but:

  1. The full sha1 is more annoying to work with. You'd have to cut and
     paste or otherwise script getting it to fetch.  A human-readable
     ref, though, is much easier to remember. The "refs/pull/N/head"
     pattern is simple to learn and type.

  2. Related to (1) above, is that it may be easier to come up with a
     hidden ref name out of band than the full sha1. E.g., if I am
     looking at https://github.com/me/foo.git/pulls/123, I can easily
     construct the ref from that. Getting the sha1 will take extra
     steps.

  3. You have to do the out-of-band step, which may be inconvenient,
     every time the ref is updated. There is no way to say "just give me
     what is at the tip of refs/pull/123/head".

I think you could solve it by teaching upload-pack to understand refs on "want" lines and convert them into the pointed-to object.

But taking a step back, this really seems quite inferior to an extension that would allow the client to share its refspecs with the server. That would solve the bandwidth efficiency problem for normal fetchers who are looking at "refs/heads/*", while still giving people who are interested in "refs/pull/*" (or even a specific refs/pull tip) the information they need to fetch.

The obvious problem is that the server speaks first. But I recall somebody suggested a combination of:

  1. For git-over-ssh and git-over-tcp, the server advertises
     tell-me-your-refspecs as it starts advertising.  Client interrupts
     advertisement with refspecs once it sees that it is OK to do so.

     We waste some bandwidth during the round-trip, but there will still
     be a benefit for repos with many refs (I wonder if we could even
     re-order the advertisement to show refs/heads/ first, as they are
     the most likely case to be requested). And as time goes on and the
     majority of clients support tell-me-your-refspecs, the server side
     can introduce a short delay after the first advertisement.

  2. For git-over-http, the client speaks first via the http protocol.
     We can stuff the refspecs into extra query parameters.

It's a little more complicated as a solution, but I feel like it gets the efficiency without a loss of functionality. And it helps in more situations than the hidden refs proposal (e.g., fetching refs/heads/foo can avoid enumerating all of refs/heads/*).

-Peff

On Tue, Feb 05, 2013 at 04:19:38AM -0500, Jeff King wrote:

I should have read your cover letter more closely, as I see you make the point there that this is no longer about the efficiency, but about uncluttering.

I'm not sure I like it as an uncluttering tool, though; for the reasons I stated in my previous mail, it makes it much more awkward for the moments when you actually do want to fetch some of the "clutter".

-Peff

Jeff King <peff@peff.net> writes:

Naming with unadvertised *refname*, not object name, needs protocol extension for the serving side to expand the name to object name; otherwise the receiving end wouldn't know what tip what it asked resulted in.

And that belongs to a separate "expand refs" extension (aka "delaying ref advertisement") that is outside the scope of this series but can be built on top, as I said in the cover letter.

This does not work yet, and for a good reason. As the side that pushes to a hidden ref never sees that the ref already exists, a request to update such a ref will come in the form of "please _create_ this ref and point it at this object", which will not pass the compare-and-swap based anti-race safety at the receiving end.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/config.txt |  6 ++++++
 builtin/receive-pack.c   |  9 ++++++++-
 t/t5516-fetch-push.sh    | 24 ++++++++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

On 01/30/2013 07:45 PM, Junio C Hamano wrote:

I would again like to express my discomfort about this feature, which is already listed as "will merge to next". Frankly, I have the feeling that this feature is being steamrolled in before a community consensus has been reached and indeed before many valid points raised by other members of the community have even been addressed. For example:

* I didn't see a response to Peff's convincing arguments that this
should be a client-side feature rather than a server-side feature [1].

* I didn't see an answer to Duy's question [2] about what is different
between the proposed feature and gitnamespaces.

* I didn't see a response to my worries that this feature could be
abused [3].

I also think that the feature is poorly designed.  For example:

* Why should a repository have exactly one setting for what refs should
be hidden?  Wouldn't it make more sense to allow multiple "views" to be
defined?:

[view "official"]
	hiderefs = refs/pull
	hiderefs = refs/heads/??/*
[view "pu"]
	hiderefs = refs/pull
[view "current"]
	hiderefs = refs/tags/releases

with the view perhaps selected via a server-side environment variable? This would allow multiple views to be published via different URLs but referring to the same git repository.

* Is it enough to support only reference exclusion (as opposed to
exclusion and inclusion rules)?  Is it enough to support only reference
selection by hierarchy (for example, how would you hide contributed
branches from your repo)?  Can your configuration scheme be expanded in
a backwards-compatible way if these or other extensions are added later?

* Why should this feature only be available remotely?  It would be handy
to clone everything but usually only see some subset of references in my
daily work: "GIT_VIEW=official gitk --all &".  Or to hide some remote
branches most of the time without having to remove them from my repo:

[view "brief"]
	refs = refs
	refs = !refs/remotes
	refs = refs/remotes/origin
	refs = refs/remotes/my-boss

I think there are still more questions than answers about this feature and FWIW vote -1 on merging it to next at this time.

Michael

[1] http://article.gmane.org/gmane.comp.version-control.git/214168 [2] http://article.gmane.org/gmane.comp.version-control.git/214070 [3] http://article.gmane.org/gmane.comp.version-control.git/213957

Hi Michael,

Michael Haggerty wrote:

In $dayjob I work with Gerrit, so I think I can start to answer some of these questions.

The client can't control the size of the ref advertisement. That is the main motivation if I understood correctly.

Namespaces are more complicated and don't sit well in existing setups involving git repositories whose refs are not namespaced.

Can you elaborate? Do you mean that through social engineering an attacker would convince the server admin to store secrets using a hidden ref and enable the upload-archive service?

That does sound like a reasonable concern. Perhaps the documentation should be updated along these lines

	transfer.hiderefs::
		String(s) `upload-pack` and `receive-pack` use to decide
		which refs to omit from their initial advertisement.  Use
		more than one transfer.hiderefs configuration variables to
		specify multiple prefix strings. A ref that are under the
		hierarchies listed on the value of this variable is excluded,
		and is hidden from `git ls-remote`, `git fetch`, `git push :`,
		etc.  An attempt to update or delete a hidden ref by `git push`
		is rejected, and an attempt to fetch a hidden ref by `git fetch`
		will fail.
	+
	This setting does not currently affect the `upload-archive` service.

until someone interested implements the same for upload-archive.

That's another reasonable concern. It's very hard to get a design correct right away, which is presumably part of the motivation of getting this into the hands of interested users who can give feedback on it. What would potentially be worth blocking even that is concerns about the wire protocol, since it is hard to take back mistakes there.

How do I request a different view of the repository at /path/to/repo.git over the network? How can we make the common case of only one view easy to achieve? Isn't the multiple-views case exactly what gitnamespaces is for?

[...]

The motivating example is turning off advertisement of the refs/changes hierarchy. If and when more complicated cases come up, that would presumably be the time to support more complicated configuration.

[...]

It is about transport. Ref namespaces have their own set of use cases and are a distinct feature.

Hoping that clarifies, Jonathan

On 02/05/2013 09:33 AM, Jonathan Nieder wrote:

Not according to Junio [4]:

  Look at this as a mechanism for the repository owner to control the
  clutter in what is shown to the intended audience of what s/he
  publishes in the repository.  Network bandwidth reduction of
  advertisement is a side effect of clutter reduction, and not
  necessarily the primary goal.

Thanks.

The upload-archive service is not needed; after patch v3 "7/8" remote clients are able to retrieve hidden content by SHA1.

Hiderefs creates a "dark" corner of a remote git repo that can hold arbitrary content that is impossible for anybody to discover but nevertheless possible for anybody to download (if they know the name of a hidden reference). In earlier versions of the patch series I believe that it was possible to push to a hidden reference hierarchy, which made it possible to upload dark content. The new version appears (from the code) to prohibit adding references in a hidden hierarchy, which would close the main loophole that I was worried about. But the documentation and the unit tests only explicitly say that updates and deletes are prohibited; nothing is said about adding references (unless "update" is understood to include "add"). I think the true behavior should be clarified and tested.

I was worried that somehow this "dark" content could be used for malicious purposes; for example, pushing compromised code then convincing somebody to download it by SHA1 with the implicit argument "it's safe since it comes directly from the project's official repository". If it is indeed impossible to populate the dark namespace remotely then I can't think of a way to exploit it.

Yes, this sounds reasonable.

Hidden references can only be configured by somebody with local access to the repository being served. Somebody with that access could also configure views. He could also probably organize a mapping from URL -> (GIT_PATH, GIT_VIEW) and offer several different views of the same repository. If the setting of environment variables is thought to sometimes be too problematic, the default view could be defined via configuration, too:

[view]
	default = official
[view "official"]
	hiderefs = refs/pull

Or perhaps, if views are made available locally too, then one view could be designated as the default for transfer purposes:

[transfer]
	defaultview = official

gitnamespaces have the disadvantage that you mentioned yourself earlier.

I suppose should the need arise we could later introduce "transfer.showrefs" and let the longest match to a given reference (i.e., "transfer.hiderefs" vs. "transfer.showrefs") win.

gitnamespaces offer multiple perspectives of a single object database, using independent sets of references to define the subset. Changing a reference in one namespace has no effect on similarly-named references in another namespace.

The "views" feature that I vaguely suggested, on the other hand, would offer multiple subsets of a single set of references, a bit like an SQL view. Changing a reference in one view would automatically affect it in other views that include the reference. It could be a handy way to reduce clutter *locally* in the same way that hiderefs would reduce clutter *remotely*. But more importantly, both features could be built on the same foundation.

Michael

[4] http://article.gmane.org/gmane.comp.version-control.git/213984

Michael Haggerty <mhagger@alum.mit.edu> writes:

See my response to Jonathan.

That is why allow-tip-sha1-in-want is a separate opt-in feature only the server side controls.

On Tue, Feb 5, 2013 at 5:29 PM, Michael Haggerty <mhagger@alum.mit.edu> wrote:

Or you can think hiderefs is the first step to addressing the initial ref advertisment problem. The series says hidden refs are to be fetched out of band, but that's not the only way. A new extension can be added to the protocol later to let the client explore this dark space. It's only truly dark for old clients. We could even shed some light to old clients by sending a dummy ref with some loud name like PLEASE_UPDATE_TO_LATEST_GIT_TO_FETCH_REMAINING_REFS (new clients silently drop this ref) -- Duy

Duy Nguyen <pclouds@gmail.com> writes:

Let me help unconfuse this thread.

I think the series as 8-patch series was poorly presented, and separating it into two will help understanding what they are about.

The first three:

  upload-pack: share more code
  upload-pack: simplify request validation
  upload/receive-pack: allow hiding ref hierarchies

is _the_ topic of the series. As far as I am concerned (I am not speaking for Gerrit users, but am speaking as the Git maintainer), the topic is solely about uncluttering. There may be refs that the server end may need to keep for its operation, but that remote users have _no_ business knowing about. Allowing the server to keep these refs in the repository, while not showing these refs over the wire, is the problem the series solves.

In other words, it is not about "these are *usually* not wanted by clients, so do not show them by default". It is about "these are not to be shown, ever".

OK?

Now, there may be some refs that are not *usually* wanted by clients but there may be cases where clients want to

 (1) learn about them via the same protocol; and/or
 (2) fetch them over the protocol.

If you want to solve both of these two issues generally, the solution has to involve a separate protocol from the today's protocol. It would go like this:

 * The upload-pack-2 service sits on a port different from today's,
   waits for a ls-remote/fetch/clone client to connect to it, makes
   a default advertisement that only includes the refs that are
   usually wanted by clients with hints on what other refs the
   initial advertisement omitted, to let the client know that it is
   allowed to ask for them.

 * An updated client, if it sees that some refs are omitted from the
   initial advertisement *and* what the user told it to fetch or
   list may be one of the omitted ones (this is why the server gives
   hints in the previous step in the first step; when the server
   says it did not omit anything, or when it says it omitted only
   refs/pull/*, a client that wanted to fetch refs/heads/frotz will
   know the request will fail without continuing this step), then
   makes a "expand-refs" request to the server, asking for the refs
   it did not see and the server could supply.

 * When the server sees "expand-refs", it responds with additional
   advertisement.  "expand-refs refs/pull/*" may result in listing
   of all refs in that hierarchy.  "expand-refs refs/changes/1/1"
   would result in listing that single ref.  "expand-refs no-such"
   may result in nothing, indicating an error.

 * After the (possible) expand-refs exchange, the client knows
   exactly the same and necessary information as the current
   protocol gives it in order to go to the common ancestor discovery
   step, and the protocol can continue the same way as the current
   protocol.

Note that this cannot sit on the current port in general, as existing clients will not be able to tell some refs are not advertised, so unless you are hiding large and truly unused part of the refspace, interoperability with older clients will render the mechanism useless. You cannot use this to delay the refs/tags/ hierarchy with this mechanism and have older client come to the updated service that by default does not advertise tags, for example.

The above is what I called the "delayed advertisement" in the discussion, which was brought up several months ago but nothing materialized as the result. People who are interested in pursuing this can volunteer and start discussing the design refinements now and submit implementation for reviews.

But in the meantime, if there is a niche use case where a solution to only the second problem is sufficient (and Gerrit and GitHub pull requests could both be such use cases), the remainder of the series can help, without waiting the solution to solve "usually not wanted but may need to be learned" problem. That is the latter 4 patches (the very last one is a demonstration to illustrate why allowing a push to hidden ref hierarchy would not and should not work, and is not for application):

  parse_fetch_refspec(): clarify the codeflow a bit
  fetch: use struct ref to represent refs to be fetched
  upload-pack: optionally allow fetching from the tips of hidden refs
  fetch: fetch objects by their exact SHA-1 object names

Junio C Hamano wrote:

[...]

An obvious question when looking at that alone is, is there ever actually need for such private refs? If the refs are not meant to be shared with users *at all*, why are they even refs?

An answer is "because refs force gc to keep the corresponding objects". For example, the sysadmin may want to keep refs/archived/ refs for dead branches that should not be advertised or accessible to the user any more. Seems sane, though not especially exciting.

What is more exciting to me is that it is a first step toward addressing the complicated problem of offering access to more refs than can be efficiently presented in the current ref advertisement. I think that's a harder problem but something like this would be needed in order to support existing clients without performance degredation.

And in the meantime, it helps with the refs/archived case.

Thanks for explaining. Jonathan

On 02/06/2013 08:17 PM, Junio C Hamano wrote:

Yes, the first three patches sound much more reasonable if this is the goal. Do you know of users who want the feature defined by the first three patches, or is it only a stepping stone towards an actually useful feature? (I ask because I have trouble imagining a real-world scenario where these alone would be useful.)

[... omitted clear explanation of how delayed advertisement could be implemented via a new protocol ...]

Given that some people *do* want to fetch all pull requests, is this a feature that any hosting service would really turn on? True, the majority of users would be spared clutter, but at the cost of completely preventing other users from fetching all pull requests, mirroring the repository, etc.

In other words, I wonder whether your two incremental steps are useful
at all, in the real world, without yet-to-be-implemented future changes.
 If not, then it doesn't make sense to merge them without at least
imagining the final goal and gaining confidence that they are not false
starts.

I think that a more useful interim solution would be to make it easy to
have two URLs accessing a single git repository, with different levels
of reference visibility applied to each.  This is something that
providers could turn on without sacrificing any existing functionality.
 And it would solve all three problems: clutter, bandwidth, and provenance.

Your first three patches would allow two-tier access to be implemented, for example by setting GIT_CONFIG or GIT_CONFIG_PARAMETERS or command-line parameters differently for the processes serving the two URLs, like:

    git upload-pack ...

vs.

    GIT_CONFIG=config-with-hidden-refs git upload-pack ...
or
    git -c transfer.hiderefs=refs/pull upload-pack ...

But this is a bit awkward because the admin would either have to maintain two config files, or maintain the hiderefs configuration in the script starting upload-pack rather than in the configuration file.

Therefore, I suggest a slight change to how hiderefs are configured to make two-tier URLs easier to configure, such as

    # Define one or more views:
    [view "uncluttered"]
            hiderefs = refs/pull

    # This would set the default view for all services:
    [transfer]
            view = uncluttered

    # Peff also wanted the possibility to configure each service
    # independently which could be done like this:
    [receive]
            view = uncluttered
    [uploadpack]
            view = full

I also tentatively suggest that we add a git-level option "--view" and an environment variable GIT_VIEW (similar to "--namespace" and GIT_NAMESPACE) to override the default setting:

    GIT_VIEW=uncluttered git upload-pack ...

This way whoever starts the process only needs to choose a particular view name; the actual definition would reside in the config file.

I think these changes would make it easier to support two-tier URLs and would also leave the way open to use the "view" concept for other things in the future.

I've said my piece now and am gratified that there has been more discussion about your proposal, which was my main goal. Therefore FWIW I turn my -1 into a -0 and leave it up to the people experiencing more clutter-induced pain to decide how to proceed.

Michael

Michael Haggerty <mhagger@alum.mit.edu> writes:

I think you said "more reasonable" without understanding what you are saying is "more reasonable", then. The mechanism is about server side wanting to use refs to protect its own metadata from gc without having to expose them; there is no "different levels".

It is awkward to use hammer to drive screws in wood, too. You want to use a screwdriver. The first three patches are to drive a nail with hammer, OK? Screws you keep bringing up is to be handled by delayed ref advertisement.

On Wed, Feb 6, 2013 at 8:17 PM, Junio C Hamano <gitster@pobox.com> wrote:

Maybe this should be split up into a different thread, but:

I think there's a simpler way to do this, which is that:

 * New clients supporting v2 of the protocol send some piece of data
   that would break old servers.

 * If that fails the new client goes "oh jeeze, I guess it's an old
   server", and try again with the old protocol.

 * The client then saves a date (or the version the server gave us)
   indicating that it tried the new protocol on that remote, tries
   again sometime later.

We already covered in previous discussions how this would be simpler with the HTTP protocol, since you could just send an extra header inviting the server to speak the new protocol.

But for the other transports we can just try the new protocol and retry with the old one as a fallback if it doesn't work. That'll allow us to gracefully migrate without needing to change the git:// port.

Besides, I think the vast majority of users are using Git via http:// or ssh://, where we can't just change the port, but even so making people change the port when we could handle this more gracefully would be a big PITA. Adding new firewall holes is often a big bureaucratic nightmare in some organizations.

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

For that to work, the new server needs to wait for the client to speak first. How would that server handle old clients who expect to be spoken first? Wait with a read timeout (no timeout is the right timeout for everybody)?

Yes, that is a solved problem.

On Wed, Feb 06, 2013 at 04:12:10PM -0800, Junio C Hamano wrote:

If the new client can handle the old-style server's response, then the server can start blasting out refs (optionally after a timeout) and stop when the client interrupts with "hey, wait, I can speak the new protocol". The server just has to include "you can interrupt me" in its capability advertisement (obviously it would have to send out at least the first ref with the capabilities before the timeout).

-Peff

On Thu, Feb 7, 2013 at 1:16 AM, Jeff King <peff@peff.net> wrote:

Can't this also be handled by passing an extra argument to upload-pack? Whether you're talking http, ssh + normal shell, ssh + git-shell or git:// you pass some argument that older clients would reject on but would cause newer clients that know about that argument to wait for you to speak before blasting refs at you.

It would mean that older clients (e.g. older git-shell) would reject your initial connection, but you could just try again, and save away info about that remote's version.

Jeff King <peff@peff.net> writes:

Yeah, I would prefer people to come up with a way to share the port and autodetect. It is *not* a requirement for the updated server to run on a separate port at all.

(Digging up an old thread about initial refs listing in git protocol)

On Thu, Feb 7, 2013 at 7:12 AM, Junio C Hamano <gitster@pobox.com> wrote:

I think the client always speaks first when it asks for a remote service. Earlier in this thread you described the new protocol upload-pack-2. Why can't it be a new service "upload-pack-2" in git-daemon?

So new client will try requesting "upload-pack-2" service with client capability advertisement before ref listing. Old servers do not recognize this service and disconnect so the new client falls back to the good old "upload-pack" (one more round trip though, but you could configure new client to use old protocol for certain "old" hosts). Similar thing happens for ssh transport. "upload-pack" service is always there for old clients.

On Sun, Feb 23, 2014 at 09:44:14AM +0700, Duy Nguyen wrote:

And now I am responding to it slowly. :)

Right, I recall the general feeling being that such a system would work, and the transition would be managed by a config variable like "remote.*.useUploadPack2". Probably with settings like:

  true:
    always try, but allow fall back to upload-pack

  false:
    never try, always use upload-pack

  auto:
    try, but if we fail, set remote.*.uploadPackTimestamp, and do not
    try again for N days

The default would start at false, and people who know their server is very up-to-date can turn it on. And then when many server implementations support it, flip the default to auto. And either leave it there forever, or eventually just set it to "true" and drop "auto" entirely as a code cleanup.

In theory we could do more radical protocol changes here, but I think most people are just interested in adding an opportunity for the client to speak before the ref advertisement in order to set a few flags/variables. That should be relatively simple, and for http we can probably pass those flags via url parameters without any extra compatibility/round-trip at all.

I think the main flag of interest is giving an fnmatch pattern to limit the advertised refs. There could potentially be others, but I do not know of any offhand.

-Peff

Jeff King <peff@peff.net> writes:

One thing that comes to mind is where symrefs point at, which we failed to add the last time around because we ran out of the hidden-space behind NUL.

On Tue, Mar 11, 2014 at 12:32:37PM -0700, Junio C Hamano wrote:

Yeah, good idea. I might be misremembering some complications, but we can probably do it with:

  1. Teach the client to send an "advertise-symrefs" flag before the ref
     advertisement.

  2. Teach the server to include symrefs in the ref advertisement; we
     can invent a new syntax because we know the client has asked for
     it.

That does not have to come immediately, though. Done correctly, upload-pack2 is not about implementing the fnmatch feature, but allowing arbitrary capability strings from the client before the ref advertisement starts. So this just becomes an extension that we can add and advertise during that new phase.

-Peff

Jeff King <peff@peff.net> writes:

I was thinking more about the underlying protocol, not advertisement in particular, and I think we came to the same conclusion.

The capability advertisement deserves to have its own separate packet message type, when both sides say that they understand it, so that we do not have to be limited by the pkt-line length limit. We could do one message per capability, and at the same time can lift the traditional "capability hidden after the NUL is purged every time, so we need to repeat them if we want to later change it, because that is how older clients and servers use that information" insanity, for example.

On Tue, Mar 11, 2014 at 01:25:23PM -0700, Junio C Hamano wrote:

So this may be entering the "more radical changes" realm I mentioned earlier.

If the client is limited to setting a few flags, then something like http can get away with:

  GET foo.git/info/refs?service=git-upload-pack&advertise-symrefs&refspec=refs/heads/*

And it does not need to worry about upload-pack2 at all. Either the server recognizes and acts on them, or it ignores them.

But given that we do not have such a magic out-of-band method for passing values over ssh and git, maybe it is not worth worrying about. Http can move to upload-pack2 along with the rest.

One thing that _is_ worth considering for http is how the protocol starts. We do not want to introduce an extra http round-trip to the protocol if we can help it. If the initial GET becomes a POST, then it could pass along the pkt-line of client capabilities with the initial request, and the server would respond with the ref advertisement as usual.

-Peff

On Wed, Mar 12, 2014 at 3:36 AM, Jeff King <peff@peff.net> wrote:

git could go the same if we lift the restriction in 73bb33a (daemon: Strictly parse the "extra arg" part of the command - 2009-06-04). It's been five years. Old daemons hopefully have all died out by now. For ssh, I suppose upload-pack and receive-pack can take an extra argument like "advertise-symrefs&refspec=refs/heads/*" (daemon would use it too to pass the advertiment to upload-pack and receive-pack).

That would make all three not need to change the underlying protocol for capability advertisement. Old git-daemon, upload-pack and receive-pack will fail hard on the new advertisement though, unlike http. But that's no worse than upload-pack2.

Or maybe http may lead the rest to another way.

On Fri, Mar 14, 2014 at 5:37 AM, Duy Nguyen <pclouds@gmail.com> wrote:

Heh. IIRC you are talking about the DoS attack for git-daemon where you send an extra header and the process infinite loops forever? We really don't want a modern client attempting to upgrade the protocol with an ancient daemon to DoS attack that server.

You missed the SSH case. It doesn't have this slot to hide the data into.

On Fri, Mar 14, 2014 at 11:45 PM, Shawn Pearce <spearce@spearce.org> wrote:

Shouldn't vulnerable daemons be upgraded anyway? If they keep using the vulnerable version for all these 5 years, I feel no sorry for new clients DoSing them. Jeff's idea about "remote.*.useUploadPack2" still applies here so after we attack the server once, it'll be black listed for a while (or forever).

Right now we run this for ssh case: "ssh <host> git-upload-pack <repo-path>". New client can do this instead

ssh <host> git-upload-pack <repo-path> <client capability flags>

On Fri, Mar 14, 2014 at 4:30 PM, Duy Nguyen <pclouds@gmail.com> wrote:

Older servers will fail on this command, and the client must reconnect over SSH, which may mean supplying their password/passphrase again. But its remembered that the uploadPack2 didn't work so this can be blacklisted and not retried for a while.

On Fri, Mar 14, 2014 at 05:09:45PM -0700, Shawn Pearce wrote:

I wonder if we could use the environment for optional values. E.g., can we run:

  ssh host GIT_CAPABILITIES=... git-upload-pack <repo-path>

That will not work everywhere, of course. Sites with git-shell will fail, as will sites with custom ssh handler (GitHub, for example, and I imagine Gerrit sites, if they support ssh). So we'd still need some fallback, but it would work out-of-the-box in a reasonable number of cases (and it is really not that different than the http case, which is just stuffing the values into $QUERY_STRING anyway :) ).

-Peff

On Tue, Mar 18, 2014 at 12:17:39AM -0400, Jeff King wrote:

Aggressively gc'ing linux-2.6 takes forever (and it's being timed so I can't really do any heavy lifting), so I outlined what the new protocol would be instead.

Note that at least for upload-pack client capabilities can be advertised twice: the first time at transport connection level, the second time in the first "want", like in v1. I think this will keep the code change down when we have to support both protocols. Moving all capabilities to the first negotiation may touch many places, but that's for now a baseless guess.

The new capability negotiation is also added for push. We didn't pay much attention to it so far.

I thought about "GIT_CAPABILITIES= git-upload-pack ..." (and actually added it in pack-protocol.txt then deleted). The thing is, if you want to new upload-pack, you would need new git-upload-pack at the remote end that must understand "git-upload-pack <repo> <caps>" already. Making it aware about GIT_CAPABILITIES is extra cost for nothing. And we have to update git-shell to support it eventually.

Well, the "must understand" part is not entirely true. If you make git-daemon pass the early capabilities via GIT_CAPABILITIES too, upload-pack does not have to support "<repo> <caps>" syntax. The upside is if old git-upload-pack ignores this GIT_CAPABILITIES, it'll break the protocol (see below) and can print friendly error messages. git-daemon has no way of printing friendly messages because it can't negotiate side-band.

I'm still not sure. But we should support either way, not both. Anyway the text for new protocols:

-- 8< --

On Tue, Mar 18, 2014 at 9:27 PM, Duy Nguyen <pclouds@gmail.com> wrote:

I should have read my mail one more time before sending. The "git-upload-pack ignores..." sentence is wrong. If it's old, its behavior is fixed and it cannot not send or do anything new.

But on the other hand, this is good. The new protocol expects upload-pack to send its caps in a new pkt-line. The old upload-pack does not follow this, which should be the indicator for the client that this server does not support v2, so it could fall back to v1 gracefully. git:// still fails hard because git-daemon is likely old too and rejects it from the beginning. But ssh:// (without git-shell) should work, http:// too. This is a very good point for GIT_CAPABILITIES.

On Tue, Mar 11, 2014 at 8:49 AM, Jeff King <peff@peff.net> wrote:

I would add that upload-pack also advertises about the availability of upload-pack2 and the client may set the remote.*.useUploadPack2 to either yes or auto so next time upload-pack2 will be used.

On Sat, Mar 15, 2014 at 08:23:08AM +0700, Duy Nguyen wrote:

Good idea. If our auto probe is "try 1, learn to upgrade to 2 for next time", we do not have to be so conservative about flipping it on (as compared to my "try 2, fall back to 1").

-Peff

On Wed, Feb 06, 2013 at 11:17:06AM -0800, Junio C Hamano wrote:

Right. I am not opposed to this series, as it does have a use-case. And if it helps Gerrit folks or other users unclutter, great. The fact that I could throw away the custom receive.hiderefs patch we use at GitHub is a bonus. If people want fancier things, they can do them separately.

_But_. As a potential user of the feature (to hide refs/pull/*), I do not think it is sufficiently flexible for me to use transfer.hiderefs (or uploadpack.hiderefs). We use "fetch" internally to migrate objects between forks and our alternates repos. And in that case, we really do want to see all refs. In other words, all fetches are not the same: we would want upload-pack to understand the difference between a client fetch and an internal administrative fetch. But this feature does not provide that lee-way. Even if you tried:

  git fetch -u 'git -c uploadpack.hiderefs= upload-pack'

the list nature of the config variable means you cannot reset it.

This isn't a show-stopper for the series; it may just mean that it is not a good fit for GitHub's use case, but others (like Gerrit) may benefit. But since refs/pull is used as an example of where this could be applied, I wanted to point out that it does not achieve that goal.

-Peff

Jonathan Nieder <jrnieder@gmail.com> writes:

The answer to this question is more nuanced.

With the current protocol, it is upload-pack who speaks first, so there is no way for the requestor to say "I am from an updated Git suite and understand how to tell you to give me limited set of refs", before upload-pack blasts 4MB of ref advertisement to it.

If the side that fetches is potentially interested in finding out any and all refs, then an alternative solution would be to break the current protocol, open a separate port and have upload-pack-2 listen to it, sit silently to let the requestor speak first when it gets connection to that port.

But if the primary thing you are interested in is to hide the references that:

 (1) the server side needs to keep track of for its own use; but
 (2) the requestors do not have to learn about from upload-pack,

we can do so without breaking older requestors. That is what the early part of this series is about. We can view the last patch to add the allow-tip-sha1-in-want as an icing on the cake.

It has the side effect of reducing the transfer overhead, because by hiding the internal refs, the server side will stop blasting 4MB of ref advertisements the requestors are not interested in, and that would be the primary observable outcome from the end-user's point of view (i.e. your "git pull --ff-only" will become a lot faster).

Michael Haggerty <mhagger@alum.mit.edu> writes:

Do not take "will merge to next" too literally. One major purpose of marking a topic as such is exactly to solicit comments like this ;-)

Uncluttering is not about a choice client should make. "delayed advertisement" is an orthogonal issue and requires a larger protocol update (it needs to make "git fetch" speak first instead of the current protocol in which "upload-pack" speaks first).

I think Jonathan addressed this already.

You can choose not to advertise allow-tip-sha1-in-want capability; I do not think it is making things worse than the status quo.

You are welcome to extend to have different views, but how would your clients express which view they would want?

Giving a single view that the serving end decides gives us an immediate benefit of showing an uncluttered set of refs of server's choice, without making the problem space larger than necessary.

Again, I do not think you cannot extend it to do positive and negative filtering "exclude these, but include those even though they match the 'exclude these' patterns I gave you earlier".

The whole point is to give the server side a choice to show selected refs, so that it can use hidden portion for its own use. These refs should not be hidden from local operations like "gc".

I appreciate the comments, but I do not think any point you raised in this message is very much relevant as objections.

On 02/05/2013 06:27 PM, Junio C Hamano wrote:

I take "will merge to next" pretty seriously, because I know how hard it is to get *my* patch series to this state :-)

There seem to be a few issues mixed up in this topic. It is hard to reason about your patch series without understanding which scenarios and problems it is meant to address. First the problems that we might like to solve:

Clutter: The typical user is subjected to much unneeded clutter in the
         form of references that he/she will likely never use.

Bandwidth: Interactions with the remote repo (clone, fetch, etc) are
           slowed down by the large volume of unnecessary data.

Provenance: Users mistakenly think that content originates with the
            repository owner whereas it in fact came from some other
            (perhaps untrusted) source.

Now, what are some use-case scenarios in which these problems arise? As I understand it, there are a few:

Scenario 1: Some providers junk up their users' repositories with content that is not created by the repository's owner and that the owner doesn't want to appear to vouch for (e.g., GitHub pull requests). These references might sometimes be useful to fetch, singly or in bulk.

Scenario 2: Some systems junk up their users' repositories with additional references that are not interesting to most pullers (e.g., Gerrit activity markers) though they don't add questionable content.

Scenario 3: Some repository owners might *themselves* want to push references to their repository but hide them from most users (e.g., Junio's topic branches) or make them completely hidden from the rest of the world (e.g., proprietary vs. open-source branches).

In most of these cases, it would be desirable for at least some users to be able to fetch and/or push hidden content.

A first weakness of your proposal is that even though the hidden refs are (optionally) fetchable, there is *no* way to discover them remotely or to bulk-download them; they would have to be retrieved one by one using out-of-band information. And if I understand correctly, there would be no way to push hidden references remotely (whether manually or from some automated process). Such processes would have to be local to the machine holding the repository.

A second weakness of your proposal is that the repository owner would *anyway* need local access to the repo server or the help of the provider to implement reference hiding (since hidden references cannot be configured remotely). Who will choose what references to hide? Most likely each provider will pick a one-size-fits-all configuration and apply it to all of the repos that they manage. All users would be at the mercy of their provider to make wise choices and would not be able to override the choice via their client.

A third weakness of your hidden references proposal is that it is schizophrenic: some references are hidden and undiscoverable, but their content can nevertheless be made fetchable if the user happens to know the SHA1. This is more complicated to understand and reason about than the rule "exactly the content that is referred to by published references is fetchable".

What would be a better way? Providers could expose multiple views of the same repository; for example, one view with just the uncluttered content, and a second view that includes *all* fetchable references. Accessing the repository via the first view would give all of the benefits provided by your hidden reference proposal. Accessing it via the second view would allow the hidden references to be fetched (even in bulk) using purely git tools. The documentation for the second view could explain that it contains un-vetted content.

But your proposal does not admit two-tiered access to a single repository. You only support one hidden reference configuration that is applied to all remote access [1]. See below for more ideas about implementing multiple views.

Yes, if the feature is turned off then it is not worse than the status quo. But what if the feature is turned on?

Actually, I'm still not clear about how these hidden references are supposed to be created. I know that you would forbid updating or deleting hidden references via the remote protocol, but would you allow them to be created? If so, then it seems that any pusher can create dark content. Or can they only be created via a separate, local channel to the repository? In this case, it seems rather limiting that any process that wants to create hidden references has to be local.

There are several possibilities:

1. Assuming the cooperation of the provider, the provider could offer
two separate URLs: one for the uncluttered view and one for the
cluttered view.  The client would choose the view by choosing which URL
to clone from.  On the provider side, both of these URLs could refer to
the same Git repository but, for example, set an environment variable
GIT_VIEW differently depending on which URL was used.  This approach
would solve clutter, bandwidth and provenance but require cooperation
from the provider.

2a. Assuming no cooperation from the provider, the git client could have options like "git fetch --view=uncluttered URL". This would receive all references from the server but discard any that are not included in the client's "uncluttered" view definition. This would solve clutter.

2b. Again assuming no cooperation from the provider, the user could clone all references from the remote repo, but define a local "uncluttered" view that hides the extra references on the local side. The view could be selected by setting a local environment variable GIT_VIEW or via configuration option "git config view.default uncluttered". This would solve clutter in a more flexible way because the clutter would still be available locally for those occasions when the user wants to see it.

Please note that none of the above options require a new remote protocol.

If/when a new protocol is implemented, then the client could tell the server what view it wants and the server would only advertise those refs to the client:

3a. The client could tell the host what reference namespaces it wants to fetch. Its choice would only be used for the single transaction and would not be recorded on the server side.

3b. The client could pick a server-defined view by name. The server would look up the name in its own configuration to translate it into a subset of references. The views that a particular server supports would be documented in the same place that the URL is documented and might also be queryable by the client. There should probably be some standard views like "default" and "full" that every server would be expected to implement. Please note that this method can fall back to 2a when communicating with a server that does not support the new protocol.

Certainly they shouldn't be hidden from "gc", but it would be useful to be able to hide references from user-facing commands like "log --all", "log --decorate", "gitk", "grep --all" etc. For example, here are some more scenarios where clutter is annoying:

Scenario 4: I occasionally share with colleague Foo, so I want to configure his repo as a remote for mine and fetch his latest work:

    git remote add foo $URL
    git fetch foo

But now every time I do a "gitk --all" or "git log --decorate", the output is cluttered with all of his references (most of which are just old versions of references from the upstream repository that we both use). I would like to be able to hide his references most of the time but turn them back on when I need them.

Scenario 5: Our upstream repository has gazillions of release tags under "refs/tags/releases/...", sometimes including customer-specific releases. In my daily life these are just clutter. (This scenario is made worse by the fact that AFAIK there is no way to tell Git to fetch some tags but not others others.) But sometimes I need to track down a bug in a particular release and need to access that release tag. So it would be nice to be able to hide and unhide them locally.

Tl;dr summary:

* Hidden refs don't give a way to offer two-tiered remote access to a
  repository (e.g., one uncluttered view and one full view), so

  * local access to the repository would (apparently) be required to
    put *anything* in the hidden namespaces.

  * they don't help in any scenario where you *sometimes*
    want to bulk fetch the hidden refs, and even make it awkward to
    fetch single hidden refs.

* Hidden refs introduce a confusing schizophrenia between "advertised"
  and "not advertised but nonetheless fetchable".

* Hidden refs require the cooperation of the provider to configure and
  will therefore be unusable by many repository owners.

* Some small improvements (e.g. allowing *multiple* views to be
  defined) would provide much more benefit for about the same effort,
  and would be a better base for building other features in the future
  (e.g., local views).

Thanks for listening. Michael

[1] Theoretically one could support multiple views of a single repository by using something like "GIT_CONFIG=view_1_config git upload-pack ..." or "git -c transfer.hiderefs=... git upload-pack ...", but this would be awkward.

Michael Haggerty wrote:

Actually Gerrit's refs/changes refs are pretty similar to Github's refs/pull. Both are requests for code review.

[...]

For both of these use cases, putting the refs somewhere other than refs/heads, refs/tags, and refs/remotes should be enough to avoid clutter.

I agree that a --decorate-glob along the lines of "git rev-parse"'s --glob would be nice.

[...]

Would advertising GIT_CONFIG_PARAMETERS and giving examples for server admins to set it in inetd et al to provide different kinds of access to a same repository through different URLs work?

Ah, I missed this comment before. What's awkward about that? I think it's a clean way to make many aspects of how a repository is presented (including hook actions) configurable.

Thanks for your help clarifying this feature. Hopefully some of the discussion will filter into the documentation.

Jonathan

On 02/06/2013 08:55 PM, Jonathan Nieder wrote:

Thanks, yes, for release tags in particular your suggestion might be workable. But I also like the idea of being able to turn subsets of references on and off easily, and have the choice persist until I change it.

Awkwardness using GIT_CONFIG: the admin would have to maintain two separate config files with mostly overlapping content.

Awkwardness using GIT_CONFIG_PARAMETERS or "-c transfer.hiderefs=...": the hiderefs configuration would have to be maintained in some Apache config or inetd or ... (or multiple places!) rather than in the repository's config file, where it belongs.

Additional awkwardness using "-c transfer.hiderefs=...": AFAIK there is no way to turn *off* a configuration variable via a command-line option.

It's all doable, but I find it awkward.

Michael

Michael Haggerty <mhagger@alum.mit.edu> writes:

I'm the author of git-fat [1], a smudge/clean filter system for managing large files. I currently store files in the file system (.git/fat/objects) and transfer them via rsync because I want to be able to transfer exact subsets requested by the user. I would like to put this data in a git repository so that I can take advantage of packfile compression when applicable and so that I can use existing access control, but I would need to store a separate reference to each blob (so that I can transfer exact subsets). My refs would be named like 'fat-<SHA1_OF_SMUDGED_DATA>' and are known on the client side because they are in the cleaned blob (which contains only this SHA1 and the number of bytes [2]).

I believe that my use case would be well supported if git could push and pull unadvertised refs, as long as basic operations were not slowed down by the existence of a very large number of such refs.

[1] https://github.com/jedbrown/git-fat

[2] We could eliminate the performance problem of needing to buffer the entire file if the smudge filter could be passed the object size as an argument and if we could forward that size in a stream to 'git hash-object --stdin'.

Jed Brown <jed@59A2.org> writes:

I am not sure about "pushing" part, but the jc/fetch-raw-sha1 topic (split from the main jc/hidden-refs topic) should allow your script, after the client learns the set of smudged object names, to ask for

    git fetch $there $sha1_1 $sha1_2 ...

or

    git fetch $there $sha1_1:refs/fat/$sha1_1 $sha1_2:refs/fat/$sha1_2 ...

I think.

Junio C Hamano <gitster@pobox.com> writes:

Well, my out-of-band knowledge is currently the sha1 of the data contained in the blob I want, not the blob sha1 itself [1]. After experimenting with jc/hidden-refs, I think it already does exactly what I want. Specifically, I set this on the server

  git config uploadpack.hiderefs refs/fat

so that 'git ls-remote' no longer transfers these refs. Then on the client, I do

  contentid=$(sha1sum thefile | cut -f1 -d \ )
  blobid=$(git hash-object -w thefile)
  git update-ref refs/fat/$contentid $blobid

  .... more like this

  git push the-remote refs/fat/$contentid ...

and later, I can fetch specific refs using

  git fetch the-remote refs/fat/$wanted:refs/fat/$wanted ...

The client knows the desired refs out-of-band so this looks okay. It would be convenient to have '--stdin' options to 'git push' and 'git fetch'. Would a patch for that be welcome?

[1] The reason for using $contentid instead of $blobid in the key here is to avoid etching the backend=git detail into the cleaned commits.