threads / discuss / 4528

Re: 2.6.17-rc6-mm2

Subject: Re: 2.6.17-rc6-mm2

## tl;dr

11 messages between Jun 16, 2006 and Jun 20, 2006.

replies: 10people: 7as markdown or json

Goo GGooo· Jun 16, 2006, 01:14 UTC · lore
On 6/15/06, Goo GGooo <googgooo@gmail.com> wrote:
Show 16 quoted lines
> Andrew Morton wrote:
>
> > - To fetch an -mm tree using git, use (for example)
> >
> >  git fetch git://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git
> > v2.6.16-rc2-mm1
>
> I'm not able to get -mm tree from GIT. In
> http://git.kernel.org/.../smurf/linux-trees.git/refs/tags/ I can see
> the most recent tags like v2.6.17-rc6-mm2 but cg-clone
> http://git.kernel.org/.../smurf/linux-trees.git gives me only
> 2.6.16-rc3 :(
>
> I tried "cg-fetch v2.6.17-rc6-mm2" which seemed to fetch some more
> tags, then played with git-checkout & friends but still can't get the
> most recent source tree.
All right, finally this worked out:
git pull rsync://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git \
      tag v2.6.17-rc6-mm2

Strange enough with http:// instead of rsync:// I got some message about nonexistent tag.

Now when I try git pull with http:// again it says the tree is up to date. However with git:// it started downloading more things and tags.

That's confusing - I believed all protocols should behave the same way...?
Goo
Linus Torvalds· Jun 16, 2006, 02:46 UTC · re: Goo GGooo · lore
On Fri, 16 Jun 2006, Goo GGooo wrote:
> 
> That's confusing - I believed all protocols should behave the same way...?

Not really. The primary protocol is the native git one, and the others try to do a best effort, but the http protocol really can't do a very good job unless the server side has run "git update-server-info" to help the http client along.

I suspect that the -mm git tree simply doesn't do that. In fact, even the main tree didn't use to do it, but I finally just broke down and added the proper hook to make it always do it automatically when I push.

(In case Andrew wants to do that, the way to do it is:
	echo -e "#!/bin/sh\nexec git-update-server-info" > hooks/post-update
	chmod +x hooks/post-update

inside the git repository - all it will do is always execute that script, and this "git-update-server-info", after you've updated the repo).

Finally, the rsync protocol just copies all objects over, and since it doesn't even know _which_ objects it is getting, it doesn't do the normal tag following that the native git protocol does.

So to recap:
 - http is fundamentally weaker, and needs some server-side help to work
 - rsync is fine for the initial clone, but doesn't actually know what 
   it's doing, so the end result can actually even be a corrupted 
   repository, because you happened to rsync just as it was updating.
 - the native git protocol generally should be considered the golden 
   standard, where the other ones are just fallbacks in case of problems 
   (like firewalls that don't let git:// through, or more commonly hosted 
   servers that don't do the git protocol at all).
Which hopefully clarifies the issue a bit.
		Linus
Goo GGooo· Jun 16, 2006, 05:49 UTC · re: Linus Torvalds · lore
On 6/16/06, Linus Torvalds <torvalds@osdl.org> wrote:
Show 11 quoted lines
> So to recap:
>  - http is fundamentally weaker, and needs some server-side help to work
>  - rsync is fine for the initial clone, but doesn't actually know what
>    it's doing, so the end result can actually even be a corrupted
>    repository, because you happened to rsync just as it was updating.
>  - the native git protocol generally should be considered the golden
>    standard, where the other ones are just fallbacks in case of problems
>    (like firewalls that don't let git:// through, or more commonly hosted
>    servers that don't do the git protocol at all).
>
> Which hopefully clarifies the issue a bit.

Thanks for explanation. Unfortunately I can't use git:// with "git pull" (at least in git-1.3.2). First it does some traffic, that suddenly stops - I guess the server starts doing *something*, perhaps preparing the update for me or whatnot. After a pretty long while it sends some more data but in the meanwhile my ADSL router dropped the NAT entry and git sits on my side waiting for data forever. Recently I tried the same on a system with direct Inet connection and that worked just fine.

I suggest adding SO_KEEPALIVE option on the git socket.
Goo
Linus Torvalds· Jun 16, 2006, 06:39 UTC · re: Goo GGooo · lore
On Fri, 16 Jun 2006, Goo GGooo wrote:
Show 5 quoted lines
> 
> Thanks for explanation. Unfortunately I can't use git:// with "git
> pull" (at least in git-1.3.2). First it does some traffic, that
> suddenly stops - I guess the server starts doing *something*, perhaps
> preparing the update for me or whatnot.

Yeah, for a big pull, the server will have to think about the objects it is going to send you.

> I suggest adding SO_KEEPALIVE option on the git socket.

Actually, the really irritating thing is that we actually generate all these nice status updates, which just makes pulling and cloning a lot more comfortable, because you actually see what is going on, and what to expect.

Except they only work over ssh, where we have a separate channel (for stderr), and with the native git protocol all that nice status work just gets flushed to /dev/null :(

Dang. It's literally the most irritating part of the thing: the protocol itself is exactly the same whether you go over ssh:// or over git://, but that visual information about what is going on is missing, and it's surprisingly important from a usability standpoint.

And in your case, the usability downside actually turned into a real accessibility bug.

Oh, well.
		Linus
H. Peter Anvin· Jun 16, 2006, 20:56 UTC · re: Linus Torvalds · lore
Linus Torvalds wrote:
Show 15 quoted lines
> 
> Actually, the really irritating thing is that we actually generate all 
> these nice status updates, which just makes pulling and cloning a lot more 
> comfortable, because you actually see what is going on, and what to 
> expect. 
> 
> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(
> 
> Dang. It's literally the most irritating part of the thing: the protocol 
> itself is exactly the same whether you go over ssh:// or over git://, but 
> that visual information about what is going on is missing, and it's 
> surprisingly important from a usability standpoint.
> 

Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of the protocol itself. After all, the protocol already does packetization, so all it needs is a reliable way to pick out the error/status packets; we could even combine that with a machine-readable code (like SMTP et al) that could get interpreted by the other side as needed.

	-hpa
Junio C Hamano· Jun 16, 2006, 22:52 UTC · re: H. Peter Anvin · lore
"H. Peter Anvin" <hpa@zytor.com> writes:
> Perhaps we shouldn't rely on stderr, and instead have a backchannel as
> part of the protocol itself.

Concurred. This was one of the thing I was planning to do anyway.

Linus Torvalds· Jun 17, 2006, 00:22 UTC · re: H. Peter Anvin · lore
On Fri, 16 Jun 2006, H. Peter Anvin wrote:
> 
> Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of
> the protocol itself.

Absolutely. I'm just irritated at myself for not going that way in the first place, but when I originally wrote it, I had my eyes on other issues, and the nice status updates got added later..

		Linus
bert hubert· Jun 16, 2006, 22:44 UTC · re: Linus Torvalds · lore
On Thu, Jun 15, 2006 at 11:39:35PM -0700, Linus Torvalds wrote:
> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(

It won't help passing firewalls one bit, but you might consider using SCTP with multiple datastreams for this - theoretically :-)

	Bert
-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
Michal Ludvig· Jun 20, 2006, 03:01 UTC · re: Linus Torvalds · lore
Linus Torvalds wrote:
Show 13 quoted lines
> 
> On Fri, 16 Jun 2006, Goo GGooo wrote:
> 
>> I suggest adding SO_KEEPALIVE option on the git socket.
> 
> Actually, the really irritating thing is that we actually generate all 
> these nice status updates, which just makes pulling and cloning a lot more 
> comfortable, because you actually see what is going on, and what to 
> expect. 
> 
> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(

OpenBSD has CVS access to their repos over SSH even for anonymous users. Could something similar be set up on git.kernel.org as well?

> And in your case, the usability downside actually turned into a real 
> accessibility bug.

Same issue here. Thanks for the hint. Attached is a patch against git 1.4.0 that solves it perfectly in my case.

Sysctl settings (for keepalive every 10 sec): net.ipv4.tcp_keepalive_intvl=10 net.ipv4.tcp_keepalive_time=10

Michal
Set SO_KEEPALIVE option on native git:// sockets.
Signed-off-by: Michal Ludvig <michal@logix.cz>
Index: git-1.4.0/connect.c
===================================================================
--- git-1.4.0.orig/connect.c
+++ git-1.4.0/connect.c
@@ -331,7 +331,7 @@ static int git_tcp_connect_sock(char *ho
 	char *colon, *end;
 	char *port = STR(DEFAULT_GIT_PORT);
 	struct addrinfo hints, *ai0, *ai;
-	int gai;
+	int gai, option;
 
 	if (host[0] == '[') {
 		end = strchr(host + 1, ']');
@@ -363,6 +363,10 @@ static int git_tcp_connect_sock(char *ho
 				ai->ai_socktype, ai->ai_protocol);
 		if (sockfd < 0)
 			continue;
+
+		option = 1;
+		setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option));
+
 		if (connect(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
 			close(sockfd);
 			sockfd = -1;
@@ -392,7 +396,7 @@ static int git_tcp_connect_sock(char *ho
 	struct hostent *he;
 	struct sockaddr_in sa;
 	char **ap;
-	unsigned int nport;
+	unsigned int nport, option;
 
 	if (host[0] == '[') {
 		end = strchr(host + 1, ']');
@@ -433,6 +437,9 @@ static int git_tcp_connect_sock(char *ho
 		sa.sin_port = htons(nport);
 		memcpy(&sa.sin_addr, *ap, he->h_length);
 
+		option = 1;
+		setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option));
+
 		if (connect(sockfd, (struct sockaddr *)&sa, sizeof sa) < 0) {
 			close(sockfd);
 			sockfd = -1;
Linus Torvalds· Jun 20, 2006, 03:22 UTC · re: Michal Ludvig · lore
On Tue, 20 Jun 2006, Michal Ludvig wrote:
> 
> OpenBSD has CVS access to their repos over SSH even for anonymous users.
> Could something similar be set up on git.kernel.org as well?

I suspect the kernel.org people would prefer not to. And I'm almost certain that others don't want to. It would really be much better if the git protocol itself just had a sideband channel. Oh, well.

			Linus
Uwe Zeisberger· Jun 16, 2006, 12:40 UTC · re: Goo GGooo · lore
Hello,
> I suggest adding SO_KEEPALIVE option on the git socket.

I suggest to do this "manually", that is send an dummy (or status) package every x seconds. Then the server could detect if a cloning client disconnected and stop generating the pack file.

(Currently I see from time to time a git server process (IIRC git-pack-objects) that creates a packfile and only when it's done fails to send it.)

Best regards Uwe

-- 
Uwe Zeisberger

http://www.google.com/search?q=30+hours+and+4+days+in+seconds

← back to recent threads