Re: git-pull from git.git - no remote ref for pu or next?

I just got this on this morning's git-fetch:

    error: no such remote ref refs/heads/pu
    error: no such remote ref refs/heads/next
    Fetch failure: git://git.kernel.org/pub/scm/git/git.git

Here's my remotes/origin:

    URL: git://git.kernel.org/pub/scm/git/git.git
    Pull: master:origin
    Pull: man:man
    Pull: html:html
    Pull: +pu:pu
    Pull: +next:next

Randal> I just got this on this morning's git-fetch:

Randal> error: no such remote ref refs/heads/pu Randal> error: no such remote ref refs/heads/next Randal> Fetch failure: git://git.kernel.org/pub/scm/git/git.git

Randal> Here's my remotes/origin:

Randal> URL: git://git.kernel.org/pub/scm/git/git.git Randal> Pull: master:origin Randal> Pull: man:man Randal> Pull: html:html Randal> Pull: +pu:pu Randal> Pull: +next:next

And then it mysteriously fixed itself a few minutes later. Is there some sort of publishing failure, or intermittent race condition?

Or is this something unique to git.git?

Or just bad electron spin or something?

Hi,

On Tue, 12 Dec 2006, Randal L. Schwartz wrote:

Congratulations!

You are experiencing the Thundering Herd Phenomenon we talked a lot about lately (the kernel.org mirroring thread).

Ciao, Dscho

Johannes> Congratulations!

Johannes> You are experiencing the Thundering Herd Phenomenon we talked a lot about Johannes> lately (the kernel.org mirroring thread).

As long as others are sharing my pain, I'll be OK. :)

On Tue, 12 Dec 2006, Randal L. Schwartz wrote:

It's mirroring.

The way that kernel.org works is that there is one master site, which is not actually running any public services at all. That's the one that people who have write access can ssh into, and rather than run any public services, it runs the security-conscious things, like the secure logins and the automated signing scripts.

The actual _public_ sites are just mirrors, with just rsync between the things. All to keep the services on the master site minimal.

But because the public sites just mirror using rsync, and aren't really aware of git repositories etc at that stage, what can happen is that a mirroring is on-going when Junio does a push, and then the changes to the "refs/" directory might get rsync'ed before the "object/" directory does, and you end up with the public sites having references to objects that don't even _exist_ on those public sites any more.

When they then run git-daemon, git-deamon will basically see a corrupt git archive, and not expose those "broken" refs at all. Which explains what you see.

And once the mirroring completes, the issue just goes away, which explains why it just magically works five minutes later.

If the public sites used git itself to synchronize git repositories, they'd never see anything like this (because git itself will only write the new refs after it has actually updated the data). But since the thing needs mirroring for non-git uses too, and since rsync generally _works_ apart from the slight race-condition issue, that's what it just uses.

Hi,

On Tue, 12 Dec 2006, Linus Torvalds wrote:

... and git would probably change the pack structure (i.e. which objects are in which packs, or even loose) which would be too bad for all those HTTP leechers ...

Ciao, Dscho

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

Well, as it is, I end up repacking my git archives on kernel.org every two weeks or so anyway, so anybody who uses stupid protocols (rsync or http) will end up downloading everything anew anyway.

And kernel.org will probably start doing automatic repacking, since the current situation just means that some people don't repack on their own, and have tens of thousands of loose objects.

You really don't want to use the non-native protocols unless you have to, or for projects that don't change.

On Tue, 12 Dec 2006, Linus Torvalds wrote:

Maybe object sharing between repos could be a good idea too? All kernel repos are likely to contain a large subset of your own so they could have a reference on it by default. That would certainly allow for better caching and less IO on the server.

On Tue, 12 Dec 2006, Nicolas Pitre wrote:

We often do. Many people have used "git clone -s -l", and as long as you repack with the "-l" flag too, the object sharing actually increases over time as the base repo gets repacked and the cloned repo keeps using the growing base pack..

Linus Torvalds wrote:

What about "logaritmic packs" idea someone (Pasky?) posted on git mailing list: pack from last week, pack from last month except last week, pack from two months, pack from four months, pack from last year...

...with *.keep to keep archive packs it can be even automated somewhat.

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

I don't see how that would be more of a concern than the current situation with occasional repacks.

Hi,

On Tue, 12 Dec 2006, Nicolas Pitre wrote:

Oh well. I did not want to get bashed for something which is probably no problem, but I suspected that the two mirror machines could get out of sync, which could well mean that the new packs would have to be downloaded _twice_. As I said, probably no problem.

But it would become a non-problem when the HTTP transport would learn to read and interpret the .idx files, basically constructing thin packs from parts of the .pack files ("Content-Range:" comes to mind)...

Ciao, Dscho

Johannes Schindelin wrote:

cURL the CLI can do this with -r/--range option, so I think that curl the library can do this too. Mind you, this is HTTP/1.1 extension (hmmm... I wonder if many sites run HTTP/1.0 only).

See also: http://www.linux.com/article.pl?sid=06/10/10/1824245

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

Sorry, far from me to sound as if I was bashing you.

In theory that should not happen since all mirrors would get the same updates in the same steps. But in practice if one mirror fails to get updated for whatever reason then the next time around it could have a bigger pack instead of two smaller ones for the same set of objects.

Woooh.

Hi,

On Tue, 12 Dec 2006, Nicolas Pitre wrote:

Does that mean "Yes, I'll do it"? ;-)

Ciao, Dscho

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

Umm. I hope it means "Woooh, that's crazy talk".

You do realize that then you need to teach the http-walker about walking the delta chain all the way up? For big pulls, you're going to be a lot _slower_ than just downloading the whole dang thing, because the delta objects are often just ~40 bytes, and you've now added a ping-pong latency for each such small transfer.

You don't need to download many such small ranges, and suddenly the few hundred ping-pongs that got you a few tens of kB of data took longer than just downloading a big stream efficiently that got you everything.

One big reason the native git protocol is efficient is that it's largely a streaming protocol (apart from the early ref-walking, but even that tries to stream as much as possible, rather than having a back-and-forth latency for each query)

Linus Torvalds wrote:

While I think the problem is much better solved by having "archive" pack(s) and "current" pack, perhaps with always sownloading the whole delta it would be feasible?

Hi,

On Tue, 12 Dec 2006, Linus Torvalds wrote:

Two points:

- For loose objects, the HTTP walker does exactly that. This is the normal 
case for "just a few objects since the last fetch". It will _never_ be the 
case for the initial clone!

- Usually, the object fetching can be parallelized, because you want 
multiple objects which are in disjunct delta chains. And for these, you 
can say something like "Content-Range: 15-31,64-79,108-135" IIRC.

You could even fetch sensible chunks, i.e. cut only at multiples of 512 to make the transport more efficient, and only fetch the parts which are _still_ missing.

So, a crazy idea, yes. But a feasible one. Just not crazy enough to be tempting for me (I use the git protocol whenever possible, too).

Ciao,

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

Absolutely not. ;-) I know next to nothing about HTTP to start with. It just looks like a crazy idea that might actually work.

On Tue, 12 Dec 2006, Linus Torvalds wrote:

Wouldn't it be a worthy goal to exclude git repos from the rsync mirroring and use git instead? The current arrangement doesn't put git in good light for the general public not reading this mailing list wrt git reliability, even if we know it is just a minor and temporary annoyance.

A failure always makes you look bad regardless of its severity.

On Tue, 12 Dec 2006, Nicolas Pitre wrote:

Well, one of the problems is simply maintenance of kernel.org.

It's just _simpler_ to use rsync for everything.

Look at the current gitweb caching discussion. Did anybody actually step up to be a gitweb maintainer on kernel.org?

Same deal. Simplicity and lack of maintenance is sometimes not just a good idea, it's a requirement.

Linus Torvalds <torvalds@osdl.org> wrote:

If kernel.org isn't using it already, I've found the --delay-updates option of rsync works reasonably well and can cut down the race-condition window. It does use more memory and disk space, however. atomic-rsync (a perl front-end distributed with the rsync source) takes even more disk space but works across an entire subdirectory all at once (with 2 renames)