threads / patch / 12359

patch, 4 partsMake git-fetch follow tags we already have objects for sooner

Subject: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

## tl;dr

7 messages between Feb 28, 2008 and Mar 2, 2008. Diffs are folded; open one to read it.

replies: 6people: 3as markdown or json

Shawn O. Pearce· Feb 28, 2008, 08:42 UTC · lore

If autofollowing of tags is enabled, we see a new tag on the remote that we don't have, and we already have the SHA-1 object that the tag is peeled to, then we can fetch the tag while we are fetching the other objects on the first connection.

This is a slight optimization for projects that have a habit of tagging a release commit after most users have already seen and downloaded that commit object through a prior fetch session. In such cases the users may still find new objects in branch heads, but the new tag will now also be part of the first pack transfer and the subsequent connection to autofollow tags is not required.

Currently git.git does not benefit from this optimization as any release usually gets a new commit at the same time that it gets a new release tag, however git-gui.git and many other projects are in the habit of tagging fairly old commits.

Users who did not already have the tagged commit still require opening a second connection to autofollow the tag, as we are unable to determine on the client side if $tag^{} will be sent to the client during the first transfer or not. Such computation must be performed on the remote side of the connection and is deferred to another series of changes.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 builtin-fetch.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
Show changes to builtin-fetch.c +6 −0
diff --git a/builtin-fetch.c b/builtin-fetch.c
index 834fbc6..a80f95b 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -101,6 +101,10 @@ static void add_merge_config(struct ref **head,
 	}
 }
 
+static void find_non_local_tags(struct transport *transport,
+			struct ref **head,
+			struct ref ***tail);
+
 static struct ref *get_ref_map(struct transport *transport,
 			       struct refspec *refs, int ref_count, int tags,
 			       int *autotags)
@@ -159,6 +163,8 @@ static struct ref *get_ref_map(struct transport *transport,
 			ref_map->merge = 1;
 		}
 	}
+	if (tags == TAGS_DEFAULT && autotags)
+		find_non_local_tags(transport, &ref_map, &tail);
 	ref_remove_duplicates(ref_map);
 
 	return ref_map;

-- 
1.5.4.3.393.g5540
Shawn O. Pearce· Feb 29, 2008, 22:34 UTC · re: Junio C Hamano · lore

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

Junio C Hamano <gitster@pobox.com> wrote:
> This is cute.  Obviously some tests need to be adjusted for this
> change, though.

I'll take a look at the current tests this weekend and see what needs to be adjusted, if anything. I'd also like to get a few tests written for this, so we are certain the optimizations are kicking in when they are supposed to be.

-- 
Shawn.
Daniel Barkalow· Feb 29, 2008, 22:39 UTC · re: Shawn O. Pearce · lore

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
Show 8 quoted lines
> Junio C Hamano <gitster@pobox.com> wrote:
> > This is cute.  Obviously some tests need to be adjusted for this
> > change, though.
> 
> I'll take a look at the current tests this weekend and see what
> needs to be adjusted, if anything.  I'd also like to get a few
> tests written for this, so we are certain the optimizations are
> kicking in when they are supposed to be.

I'd be really grateful if you came up with a good general strategy for testing that we're not doing too much work in fetching, because clone has optimizations that I need to test in a similar way, and I haven't been able to think of anything not horribly intrusive.

	-Daniel
*This .sig left intentionally blank*
Shawn O. Pearce· Mar 1, 2008, 04:25 UTC · re: Daniel Barkalow · lore

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 15 quoted lines
> On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> 
> > Junio C Hamano <gitster@pobox.com> wrote:
> > > This is cute.  Obviously some tests need to be adjusted for this
> > > change, though.
> > 
> > I'll take a look at the current tests this weekend and see what
> > needs to be adjusted, if anything.  I'd also like to get a few
> > tests written for this, so we are certain the optimizations are
> > kicking in when they are supposed to be.
> 
> I'd be really grateful if you came up with a good general strategy for 
> testing that we're not doing too much work in fetching, because clone has 
> optimizations that I need to test in a similar way, and I haven't been 
> able to think of anything not horribly intrusive.
Is this "horribly intrusive" ?
We can test it with something like this:
  $ GIT_DEBUG_SEND_PACK=1 git fetch 3>UPLOAD_LOG
  $ cat UPLOAD_LOG
  #S
  want 8e10cf4e007ad7e003463c30c34b1050b039db78 multi_ack side-band-64k thin-pack ofs-delta
  want ddfa4a33562179aca1ace2bcc662244a17d0b503
  #E
  #S
  want 3253df4d1cf6fb138b52b1938473bcfec1483223 multi_ack side-band-64k thin-pack ofs-delta
  #E

Notice we made two connections, wanting 2 things and then later only wanting 1 thing. Tag auto-following. :-)

Show changes to upload-pack.c +7 −0
diff --git a/upload-pack.c b/upload-pack.c
index b26d053..4e14020 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -444,7 +444,10 @@ static void receive_needs(void)
 	struct object_array shallows = {0, 0, NULL};
 	static char line[1000];
 	int len, depth = 0;
+	int debug_needs = !!getenv("GIT_DEBUG_SEND_PACK");
 
+	if (debug_needs)
+		write_in_full(3, "#S\n", 3);
 	for (;;) {
 		struct object *o;
 		unsigned char sha1_buf[20];
@@ -452,6 +455,8 @@ static void receive_needs(void)
 		reset_timeout();
 		if (!len)
 			break;
+		if (debug_needs)
+			write_in_full(3, line, len);
 
 		if (!prefixcmp(line, "shallow ")) {
 			unsigned char sha1[20];
@@ -507,6 +512,8 @@ static void receive_needs(void)
 			add_object_array(o, NULL, &want_obj);
 		}
 	}
+	if (debug_needs)
+		write_in_full(3, "#E\n", 3);
 	if (depth == 0 && shallows.nr == 0)
 		return;
 	if (depth > 0) {

-- 
Shawn.
Daniel Barkalow· Mar 1, 2008, 16:33 UTC · re: Shawn O. Pearce · lore

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
Show 20 quoted lines
> Daniel Barkalow <barkalow@iabervon.org> wrote:
> > On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> > 
> > > Junio C Hamano <gitster@pobox.com> wrote:
> > > > This is cute.  Obviously some tests need to be adjusted for this
> > > > change, though.
> > > 
> > > I'll take a look at the current tests this weekend and see what
> > > needs to be adjusted, if anything.  I'd also like to get a few
> > > tests written for this, so we are certain the optimizations are
> > > kicking in when they are supposed to be.
> > 
> > I'd be really grateful if you came up with a good general strategy for 
> > testing that we're not doing too much work in fetching, because clone has 
> > optimizations that I need to test in a similar way, and I haven't been 
> > able to think of anything not horribly intrusive.
> 
> Is this "horribly intrusive" ?
> 
> We can test it with something like this:

That looks suitable to me (although maybe the environment variable should contain the fd to write to?). I also want to know the "have" info, so I can verify that the client is listing things appropriately, but that's an obvious addition.

	-Daniel
*This .sig left intentionally blank*
Shawn O. Pearce· Mar 2, 2008, 14:13 UTC · re: Daniel Barkalow · lore

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 10 quoted lines
> On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> > 
> > Is this "horribly intrusive" ?
> > 
> > We can test it with something like this:
> 
> That looks suitable to me (although maybe the environment variable should 
> contain the fd to write to?). I also want to know the "have" info, so I 
> can verify that the client is listing things appropriately, but that's an 
> obvious addition.

Yea, I was thinking about that last night myself. I think the patch is still in `pu` so I will work up something for Junio to squash into it that uses the variable to specify the fd, rather than assume 3.

-- 
Shawn.

← back to recent threads