threads / rfc / 31519

RFC patchremote-helper: allow fetch to discover sha1 later

Subject: [PATCH/RFC] remote-helper: allow fetch to discover sha1 later

## tl;dr

3 messages between Sep 13, 2012 and Sep 14, 2012. Diffs are folded; open one to read it.

replies: 2people: 2as markdown or json

Devin J. Pohly· Sep 13, 2012, 10:10 UTC · lore
From: "Devin J. Pohly" <djpohly@gmail.com>
Allow a fetch-style remote helper to issue the notification
  ref <sha1> <name>
to specify a ref's value during the fetch operation.

Currently, a remote helper with the "fetch" capability cannot fetch a ref unless its sha1 was known when the "list" command was executed. This limitation is arbitrary, as the helper may eventually be able to determine the correct sha1 as it fetches objects.

Signed-off-by: Devin J. Pohly <djpohly@gmail.com>
---
Soliciting general opinions - first git patch. :)

The fetch command can be a lot more convenient than import if you're working with a remote that doesn't give you a commit and the associated objects at the same time. And there's no apparent reason that something like this isn't possible.

How it works: the old_sha1 field is currently set when the output from "list" is parsed, then not used again until after the "fetch" command completes, just before updating refs. Changing it during the fetch only affects the final value of the ref. (Forgetting to resolve a ref will result in exactly the same behavior as before: an error from check_everything_connected.)

Not sure if either or both of the warning()s should be a die() instead.
 Documentation/git-remote-helpers.txt |  8 ++++++--
 transport-helper.c                   | 24 ++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 2 deletions(-)
Show changes to 2 files +30 −2

Documentation/git-remote-helpers.txt, transport-helper.c

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index f5836e4..fb4240f 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -229,8 +229,12 @@ Supported if the helper has the "option" capability.
 	to the database.  Fetch commands are sent in a batch, one
 	per line, terminated with a blank line.
 	Outputs a single blank line when all fetch commands in the
-	same batch are complete. Only objects which were reported
-	in the ref list with a sha1 may be fetched this way.
+	same batch are complete.
++
+If the named ref was reported in the ref list without a sha1, must
+output a 'ref <sha1> <name>' line once the sha1 is known.  This is
+also required for a symref if its target did not have a sha1 in the
+list.
 +
 Optionally may output a 'lock <file>' line indicating a file under
 GIT_DIR/objects/pack which is keeping a pack until refs can be
diff --git a/transport-helper.c b/transport-helper.c
index cfe0988..6fce419 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -363,6 +363,30 @@ static int fetch_with_fetch(struct transport *transport,
 				warning("%s also locked %s", data->name, name);
 			else
 				transport->pack_lockfile = xstrdup(name);
+		} else if (!prefixcmp(buf.buf, "ref ")) {
+			const char *rest = buf.buf + 4;
+			char *eov, *eon;
+			struct ref *posn;
+
+			eov = strchr(rest, ' ');
+			if (!eov)
+				die("Malformed ref command: %s", buf.buf);
+			eon = strchr(eov + 1, ' ');
+			*eov = '\0';
+			if (eon)
+				*eon = '\0';
+			for (i = 0; i < nr_heads; i++) {
+				posn = to_fetch[i];
+				if (!strcmp(eov + 1, posn->name))
+					break;
+			}
+			if (i == nr_heads || (posn->status & REF_STATUS_UPTODATE)) {
+				warning("Ref %s is not being fetched", eov + 1);
+				continue;
+			}
+			if (!is_null_sha1(posn->old_sha1))
+				warning("Ref %s is being overwritten", eov + 1);
+			get_sha1_hex(rest, posn->old_sha1);
 		}
 		else if (!buf.len)
 			break;
-- 
1.7.12
Junio C Hamano· Sep 14, 2012, 06:10 UTC · re: Devin J. Pohly · lore

Re: [PATCH/RFC] remote-helper: allow fetch to discover sha1 later

"Devin J. Pohly" <djpohly@gmail.com> writes:
Show 10 quoted lines
> From: "Devin J. Pohly" <djpohly@gmail.com>
>
> Allow a fetch-style remote helper to issue the notification
>   ref <sha1> <name>
> to specify a ref's value during the fetch operation.
>
> Currently, a remote helper with the "fetch" capability cannot fetch a
> ref unless its sha1 was known when the "list" command was executed.
> This limitation is arbitrary, as the helper may eventually be able to
> determine the correct sha1 as it fetches objects.

While I am not fundamentally against supporting a remote helper that is not capable of saying what the object names of the tip of its histories are before doing a lot of work, I do not think it is a good idea to allow such a helper to claim that it supports "fetch" capability, for at least two reasons:

 * Being able to "list" is essential for "fetch" based helpers, I
   think, far from "arbitrary".  When running a "git fetch" against
   such a remote, we can first issue "list" to see what it has and
   avoid asking for the refs that point at the histories we already
   have (or the refs that are the same as the last time we fetched
   from the remote).  I do not offhand know if we do this kind of
   optimization, but if we don't, we should.
 * Existing versions of "git" that can drive remote helpers that
   claim to have "fetch" capability are not prepared to accept an
   unknown "refs" protocol message from the helper, so a helper
   written for your new semantics of "fetch" capability will not
   work with them.  The point of capability mechanism is to prevent
   such a compatibility issue from breaking the system, and this
   patch goes against that spirit.

Such a remote helper should not advertise to support "fetch" capability, because it does not support it. It can advertise that it supports "something else" and it is OK to make an updated Git that knows how to drive a remote helper that lack "fetch" but support that "something else" work with it.

Devin J. Pohly· Sep 14, 2012, 20:49 UTC · re: Junio C Hamano · lore

Re: [PATCH/RFC] remote-helper: allow fetch to discover sha1 later

On Thu, Sep 13, 2012 at 11:10:26PM -0700, Junio C Hamano wrote:
Show 6 quoted lines
> I do not think it is a good idea to allow such a helper to claim that
> it supports "fetch" capability, for at least two reasons:
> 
>  * Being able to "list" is essential for "fetch" based helpers.
>    think, far from "arbitrary".
>    ...

Oh, I don't mean to change the semantics of the list command at all. What I thought seemed arbitrary was limiting the *fetch* command to refs with pre-known sha1s. Any list-time optimization in place or possible for such refs wouldn't be affected.

(In light of this, specifying a new sha1 for a ref which already had one in the list should probably be an error rather than a warning. I'll update this in the next version and make it clear that a "ref" message must only be issued for refs reported without a sha1.)

Show 6 quoted lines
>  * Existing versions of "git" that can drive remote helpers that
>    claim to have "fetch" capability are not prepared to accept an
>    unknown "refs" protocol message from the helper, so a helper
>    written for your new semantics of "fetch" capability will not
>    work with them.
>    ...

I see. I didn't realize that new helper + old Git client is a supported combination. Still, hear me out.

1. A new helper must only send a "ref" message when git asks for a ref
   without a known sha1.
2. Asking for a ref without a known sha1 is unsupported, according to
   git-remote-helpers.txt: "Only objects which were reported in the ref
   list with a sha1 may be fetched [with fetch]."
3. Furthermore, asking for a ref without a known sha1 *already* breaks
   in existing versions of git with current handlers:
    $ git fetch testfetch::../git1 foo
    fatal: bad object 0000000000000000000000000000000000000000
    error: testfetch::../git1 did not send all necessary objects
   Compare - existing version of git, with a new handler:
    $ git fetch testref::../git1 foo
    warning: testref unexpectedly said: 'ref 0f31<snip> refs/heads/foo'
    fatal: bad object 0000000000000000000000000000000000000000
    error: testref::../git1 did not send all necessary objects

So the proposed change doesn't break anything that isn't already broken. :)

That said, if you're still not sold, a new capability is fine. Though I think it can be done without it, I'm certainly not opposed to adding one.

← back to recent threads