# Re: 2.6.17-rc6-mm2 11 messages from 2006-06-16 to 2006-06-20. Participants: Goo GGooo, Linus Torvalds, Uwe Zeisberger, H. Peter Anvin, bert hubert, Junio C Hamano, Michal Ludvig. Thread: https://gitlist.dev/t/4528 ## Goo GGooo, 2006-06-16 01:14 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/ef5305790606151814i252c37c4mdd005f11f06ceac%40mail.gmail.com In-Reply-To: ``` On 6/15/06, Goo GGooo wrote: > Andrew Morton wrote: > > > - To fetch an -mm tree using git, use (for example) > > > > git fetch git://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git > > v2.6.16-rc2-mm1 > > I'm not able to get -mm tree from GIT. In > http://git.kernel.org/.../smurf/linux-trees.git/refs/tags/ I can see > the most recent tags like v2.6.17-rc6-mm2 but cg-clone > http://git.kernel.org/.../smurf/linux-trees.git gives me only > 2.6.16-rc3 :( > > I tried "cg-fetch v2.6.17-rc6-mm2" which seemed to fetch some more > tags, then played with git-checkout & friends but still can't get the > most recent source tree. All right, finally this worked out: git pull rsync://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git \ tag v2.6.17-rc6-mm2 Strange enough with http:// instead of rsync:// I got some message about nonexistent tag. Now when I try git pull with http:// again it says the tree is up to date. However with git:// it started downloading more things and tags. That's confusing - I believed all protocols should behave the same way...? Goo ``` ## Linus Torvalds, 2006-06-16 02:46 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606151937360.5498%40g5.osdl.org In-Reply-To: ``` On Fri, 16 Jun 2006, Goo GGooo wrote: > > That's confusing - I believed all protocols should behave the same way...? Not really. The primary protocol is the native git one, and the others try to do a best effort, but the http protocol really can't do a very good job unless the server side has run "git update-server-info" to help the http client along. I suspect that the -mm git tree simply doesn't do that. In fact, even the main tree didn't use to do it, but I finally just broke down and added the proper hook to make it always do it automatically when I push. (In case Andrew wants to do that, the way to do it is: echo -e "#!/bin/sh\nexec git-update-server-info" > hooks/post-update chmod +x hooks/post-update inside the git repository - all it will do is always execute that script, and this "git-update-server-info", after you've updated the repo). Finally, the rsync protocol just copies all objects over, and since it doesn't even know _which_ objects it is getting, it doesn't do the normal tag following that the native git protocol does. So to recap: - http is fundamentally weaker, and needs some server-side help to work - rsync is fine for the initial clone, but doesn't actually know what it's doing, so the end result can actually even be a corrupted repository, because you happened to rsync just as it was updating. - the native git protocol generally should be considered the golden standard, where the other ones are just fallbacks in case of problems (like firewalls that don't let git:// through, or more commonly hosted servers that don't do the git protocol at all). Which hopefully clarifies the issue a bit. Linus ``` ## Goo GGooo, 2006-06-16 05:49 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/ef5305790606152249n2702873fy7b708d9c47c78470%40mail.gmail.com In-Reply-To: ``` On 6/16/06, Linus Torvalds wrote: > So to recap: > - http is fundamentally weaker, and needs some server-side help to work > - rsync is fine for the initial clone, but doesn't actually know what > it's doing, so the end result can actually even be a corrupted > repository, because you happened to rsync just as it was updating. > - the native git protocol generally should be considered the golden > standard, where the other ones are just fallbacks in case of problems > (like firewalls that don't let git:// through, or more commonly hosted > servers that don't do the git protocol at all). > > Which hopefully clarifies the issue a bit. Thanks for explanation. Unfortunately I can't use git:// with "git pull" (at least in git-1.3.2). First it does some traffic, that suddenly stops - I guess the server starts doing *something*, perhaps preparing the update for me or whatnot. After a pretty long while it sends some more data but in the meanwhile my ADSL router dropped the NAT entry and git sits on my side waiting for data forever. Recently I tried the same on a system with direct Inet connection and that worked just fine. I suggest adding SO_KEEPALIVE option on the git socket. Goo ``` ## Linus Torvalds, 2006-06-16 06:39 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606152335130.5498%40g5.osdl.org In-Reply-To: ``` On Fri, 16 Jun 2006, Goo GGooo wrote: > > Thanks for explanation. Unfortunately I can't use git:// with "git > pull" (at least in git-1.3.2). First it does some traffic, that > suddenly stops - I guess the server starts doing *something*, perhaps > preparing the update for me or whatnot. Yeah, for a big pull, the server will have to think about the objects it is going to send you. > I suggest adding SO_KEEPALIVE option on the git socket. Actually, the really irritating thing is that we actually generate all these nice status updates, which just makes pulling and cloning a lot more comfortable, because you actually see what is going on, and what to expect. Except they only work over ssh, where we have a separate channel (for stderr), and with the native git protocol all that nice status work just gets flushed to /dev/null :( Dang. It's literally the most irritating part of the thing: the protocol itself is exactly the same whether you go over ssh:// or over git://, but that visual information about what is going on is missing, and it's surprisingly important from a usability standpoint. And in your case, the usability downside actually turned into a real accessibility bug. Oh, well. Linus ``` ## Uwe Zeisberger, 2006-06-16 12:40 Subject: Re: 2.6.17-rc6-mm2 Message-ID: <20060616124010.GB13884@informatik.uni-freiburg.de> URL: https://gitlist.dev/e/20060616124010.GB13884%40informatik.uni-freiburg.de In-Reply-To: ``` Hello, > I suggest adding SO_KEEPALIVE option on the git socket. I suggest to do this "manually", that is send an dummy (or status) package every x seconds. Then the server could detect if a cloning client disconnected and stop generating the pack file. (Currently I see from time to time a git server process (IIRC git-pack-objects) that creates a packfile and only when it's done fails to send it.) Best regards Uwe -- Uwe Zeisberger http://www.google.com/search?q=30+hours+and+4+days+in+seconds ``` ## H. Peter Anvin, 2006-06-16 20:56 Subject: Re: 2.6.17-rc6-mm2 Message-ID: <44931AFD.4070809@zytor.com> URL: https://gitlist.dev/e/44931AFD.4070809%40zytor.com In-Reply-To: ``` Linus Torvalds wrote: > > Actually, the really irritating thing is that we actually generate all > these nice status updates, which just makes pulling and cloning a lot more > comfortable, because you actually see what is going on, and what to > expect. > > Except they only work over ssh, where we have a separate channel (for > stderr), and with the native git protocol all that nice status work just > gets flushed to /dev/null :( > > Dang. It's literally the most irritating part of the thing: the protocol > itself is exactly the same whether you go over ssh:// or over git://, but > that visual information about what is going on is missing, and it's > surprisingly important from a usability standpoint. > Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of the protocol itself. After all, the protocol already does packetization, so all it needs is a reliable way to pick out the error/status packets; we could even combine that with a machine-readable code (like SMTP et al) that could get interpreted by the other side as needed. -hpa ``` ## bert hubert, 2006-06-16 22:44 Subject: Re: 2.6.17-rc6-mm2 Message-ID: <20060616224406.GA10451@outpost.ds9a.nl> URL: https://gitlist.dev/e/20060616224406.GA10451%40outpost.ds9a.nl In-Reply-To: ``` On Thu, Jun 15, 2006 at 11:39:35PM -0700, Linus Torvalds wrote: > Except they only work over ssh, where we have a separate channel (for > stderr), and with the native git protocol all that nice status work just > gets flushed to /dev/null :( It won't help passing firewalls one bit, but you might consider using SCTP with multiple datastreams for this - theoretically :-) Bert -- http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services ``` ## Junio C Hamano, 2006-06-16 22:52 Subject: Re: 2.6.17-rc6-mm2 Message-ID: <7v3be4d87y.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v3be4d87y.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <44931AFD.4070809@zytor.com> ``` "H. Peter Anvin" writes: > Perhaps we shouldn't rely on stderr, and instead have a backchannel as > part of the protocol itself. Concurred. This was one of the thing I was planning to do anyway. ``` ## Linus Torvalds, 2006-06-17 00:22 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606161720590.5498%40g5.osdl.org In-Reply-To: <44931AFD.4070809@zytor.com> ``` On Fri, 16 Jun 2006, H. Peter Anvin wrote: > > Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of > the protocol itself. Absolutely. I'm just irritated at myself for not going that way in the first place, but when I originally wrote it, I had my eyes on other issues, and the nice status updates got added later.. Linus ``` ## Michal Ludvig, 2006-06-20 03:01 Subject: Re: 2.6.17-rc6-mm2 Message-ID: <44976506.8040205@logix.cz> URL: https://gitlist.dev/e/44976506.8040205%40logix.cz In-Reply-To: ``` Linus Torvalds wrote: > > On Fri, 16 Jun 2006, Goo GGooo wrote: > >> I suggest adding SO_KEEPALIVE option on the git socket. > > Actually, the really irritating thing is that we actually generate all > these nice status updates, which just makes pulling and cloning a lot more > comfortable, because you actually see what is going on, and what to > expect. > > Except they only work over ssh, where we have a separate channel (for > stderr), and with the native git protocol all that nice status work just > gets flushed to /dev/null :( OpenBSD has CVS access to their repos over SSH even for anonymous users. Could something similar be set up on git.kernel.org as well? > And in your case, the usability downside actually turned into a real > accessibility bug. Same issue here. Thanks for the hint. Attached is a patch against git 1.4.0 that solves it perfectly in my case. Sysctl settings (for keepalive every 10 sec): net.ipv4.tcp_keepalive_intvl=10 net.ipv4.tcp_keepalive_time=10 Michal Set SO_KEEPALIVE option on native git:// sockets. Signed-off-by: Michal Ludvig Index: git-1.4.0/connect.c =================================================================== --- git-1.4.0.orig/connect.c +++ git-1.4.0/connect.c @@ -331,7 +331,7 @@ static int git_tcp_connect_sock(char *ho char *colon, *end; char *port = STR(DEFAULT_GIT_PORT); struct addrinfo hints, *ai0, *ai; - int gai; + int gai, option; if (host[0] == '[') { end = strchr(host + 1, ']'); @@ -363,6 +363,10 @@ static int git_tcp_connect_sock(char *ho ai->ai_socktype, ai->ai_protocol); if (sockfd < 0) continue; + + option = 1; + setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option)); + if (connect(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { close(sockfd); sockfd = -1; @@ -392,7 +396,7 @@ static int git_tcp_connect_sock(char *ho struct hostent *he; struct sockaddr_in sa; char **ap; - unsigned int nport; + unsigned int nport, option; if (host[0] == '[') { end = strchr(host + 1, ']'); @@ -433,6 +437,9 @@ static int git_tcp_connect_sock(char *ho sa.sin_port = htons(nport); memcpy(&sa.sin_addr, *ap, he->h_length); + option = 1; + setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option)); + if (connect(sockfd, (struct sockaddr *)&sa, sizeof sa) < 0) { close(sockfd); sockfd = -1; ``` ## Linus Torvalds, 2006-06-20 03:22 Subject: Re: 2.6.17-rc6-mm2 Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606192016210.5498%40g5.osdl.org In-Reply-To: <44976506.8040205@logix.cz> ``` On Tue, 20 Jun 2006, Michal Ludvig wrote: > > OpenBSD has CVS access to their repos over SSH even for anonymous users. > Could something similar be set up on git.kernel.org as well? I suspect the kernel.org people would prefer not to. And I'm almost certain that others don't want to. It would really be much better if the git protocol itself just had a sideband channel. Oh, well. Linus ```