threads / discuss / 1188

What broke snapshots now?

Subject: What broke snapshots now?

## tl;dr

9 messages between Jul 9, 2005 and Jul 11, 2005.

replies: 8people: 3as markdown or json

David Woodhouse· Jul 9, 2005, 11:08 UTC · lore

Does git on master.kernel.org need to be updated to handle packed objects? See attached.

Linus, please could you add the snapshot script to your regression testing? http://david.woodhou.se/git-snapshot.sh

It'd be good to keep that working without too much manual intervention. 
-- 
dwmw2



+ case `hostname` in
++ hostname
+ export PATH=/home/dwmw2/cogito:/usr/bin:/bin
+ PATH=/home/dwmw2/cogito:/usr/bin:/bin
+ export BASE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
+ BASE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
+ STAGINGLOCK=/staging/upload.lock
+ FINAL=/pub/linux/kernel/v2.6/snapshots
+ '[' '!' -d /pub/scm/linux/kernel/git/torvalds/linux-2.6.git ']'
+ export WORK_DIRECTORY=/home/dwmw2/snapshots/2.6
+ WORK_DIRECTORY=/home/dwmw2/snapshots/2.6
+ export SNAP_TAG_DIRECTORY=/home/dwmw2/snapshots/2.6/tags
+ SNAP_TAG_DIRECTORY=/home/dwmw2/snapshots/2.6/tags
+ export STAGE=/home/dwmw2/snapshots/2.6/stage
+ STAGE=/home/dwmw2/snapshots/2.6/stage
+ export SHA1_FILE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects
+ SHA1_FILE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects
++ ls -rt /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.11 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.11-tree /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12-rc2 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12-rc3 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12-rc4 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12-rc5 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.12-rc6 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.13-rc1 /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.13-rc2
++ tail -n1
++ sed s:/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v::
+ RELNAME=2.6.13-rc2
++ cat /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/refs/tags/v2.6.13-rc2
+ RELOBJ=c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19
++ tail -n1
++ sed s:/home/dwmw2/snapshots/2.6/tags/v::
++ ls -rt /home/dwmw2/snapshots/2.6/tags/v2.6.13-rc2-git1
+ SNAPNAME=2.6.13-rc2-git1
+ '[' 2.6.13-rc2-git1 == '' ']'
++ cat /home/dwmw2/snapshots/2.6/tags/v2.6.13-rc2-git1
+ LASTOBJ=c101f3136cc98a003d0d16be6fab7d0d950581a6
++ echo 2.6.13-rc2-git1
++ sed 's/^.*-git//'
+ OLDGITNUM=1
++ expr 1 + 1
+ NEWGITNUM=2
+ CURNAME=2.6.13-rc2-git2
++ tree-id c101f3136cc98a003d0d16be6fab7d0d950581a6
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/c1/01f3136cc98a003d0d16be6fab7d0d950581a6: No such file or directory
fatal: git-cat-file c101f3136cc98a003d0d16be6fab7d0d950581a6: bad file
Invalid id: c101f3136cc98a003d0d16be6fab7d0d950581a6
usage: git-cat-file [-t | tagname] <sha1>
usage: git-cat-file [-t | tagname] <sha1>
Invalid id: 
+ LASTTREE=
++ cat /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/HEAD
+ CURCOMM=a92b7b80579fe68fe229892815c750f6652eb6a9
++ tree-id a92b7b80579fe68fe229892815c750f6652eb6a9
+ CURTREE=7fd73e9f39bf6003cc3188a10426b62d8c47ab40
++ tree-id c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/c5/21cb0f10ef2bf28a18e1cc8adf378ccbbe5a19: No such file or directory
fatal: git-cat-file c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19: bad file
Invalid id: c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19
usage: git-cat-file [-t | tagname] <sha1>
usage: git-cat-file [-t | tagname] <sha1>
Invalid id: 
+ RELTREE=
+ echo release 2.6.13-rc2 commit tree
release 2.6.13-rc2 commit tree
++ git-cat-file -t c101f3136cc98a003d0d16be6fab7d0d950581a6
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/c1/01f3136cc98a003d0d16be6fab7d0d950581a6: No such file or directory
fatal: git-cat-file c101f3136cc98a003d0d16be6fab7d0d950581a6: bad file
+ echo last c101f3136cc98a003d0d16be6fab7d0d950581a6 tree
last c101f3136cc98a003d0d16be6fab7d0d950581a6 tree
+ echo head a92b7b80579fe68fe229892815c750f6652eb6a9 tree 7fd73e9f39bf6003cc3188a10426b62d8c47ab40
head a92b7b80579fe68fe229892815c750f6652eb6a9 tree 7fd73e9f39bf6003cc3188a10426b62d8c47ab40
+ '[' '' == 7fd73e9f39bf6003cc3188a10426b62d8c47ab40 ']'
++ echo 2.6.13-rc2-git2
++ cut -f2- -d-
+ EXTRAVERSION=-rc2-git2
+ cd /home/dwmw2/snapshots/2.6/stage
+ rm -rf tmp-empty-tree
+ mkdir -p tmp-empty-tree/.git
+ cd tmp-empty-tree
+ git-read-tree a92b7b80579fe68fe229892815c750f6652eb6a9
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/f8/640c306db2d583b9a30f2e52f8fb0a4cf624e0: No such file or directory
fatal: failed to unpack tree object a92b7b80579fe68fe229892815c750f6652eb6a9
+ git-checkout-cache Makefile
checkout-cache: Makefile is not in the cache.
+ perl -pi -e 's/EXTRAVERSION =.*/EXTRAVERSION = -rc2-git2/' Makefile
Can't open Makefile: No such file or directory.
+ git-diff-cache -m -p
+ gzip -9
usage: diff-cache [-r] [-z] [-p] [-i] [--cached] <tree sha1>
+ echo a92b7b80579fe68fe229892815c750f6652eb6a9
+ cg-log c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19:a92b7b80579fe68fe229892815c750f6652eb6a9
mv: cannot stat `.git/heads': No such file or directory
mv: cannot stat `.git/tags': No such file or directory
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/c5/21cb0f10ef2bf28a18e1cc8adf378ccbbe5a19: No such file or directory
fatal: git-cat-file c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19: bad file
Invalid id: c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19
+ echo a92b7b80579fe68fe229892815c750f6652eb6a9
+ echo New Snapshot 2.6.13-rc2-git2
New Snapshot 2.6.13-rc2-git2
+ '[' -z /pub/linux/kernel/v2.6/snapshots ']'
+ '[' -r /staging/upload.lock ']'
+ exec
+ flock -s 200
+ mv -v /home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.gz /home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.id /home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.log /pub/linux/kernel/v2.6/snapshots
`/home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.gz' -> `/pub/linux/kernel/v2.6/snapshots/patch-2.6.13-rc2-git2.gz'
`/home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.id' -> `/pub/linux/kernel/v2.6/snapshots/patch-2.6.13-rc2-git2.id'
`/home/dwmw2/snapshots/2.6/stage/patch-2.6.13-rc2-git2.log' -> `/pub/linux/kernel/v2.6/snapshots/patch-2.6.13-rc2-git2.log'
Jeff Garzik· Jul 9, 2005, 13:37 UTC · re: David Woodhouse · lore

Re: What broke snapshots now?

David Woodhouse wrote:
> Does git on master.kernel.org need to be updated to handle packed
> objects?
Yes.

It's always fun when the on-disk format of the upstream 2.6.x kernel repo changes without notice :/

	Jeff
Linus Torvalds· Jul 9, 2005, 16:15 UTC · re: David Woodhouse · lore

Re: What broke snapshots now?

On Sat, 9 Jul 2005, David Woodhouse wrote:
>
> Does git on master.kernel.org need to be updated to handle packed
> objects? See attached.

Yes, looks that way. Except it's not "git on master.kernel.org", it's "git in your home directory", I suspect. I expressly held off packing the kernel repo until git had been updated on kernel.org.

So while the regular git on kernel.org should already be up-to-date, you have

	PATH=/home/dwmw2/cogito:/usr/bin:/bin
in your script ;)
> Linus, please could you add the snapshot script to your regression
> testing? http://david.woodhou.se/git-snapshot.sh

None of the git-specific parts should have broken - you don't even do anything strange there, I think we already have regression tests in git for everything you use. But the fact that you had an old version is not something I can do much about ;)

		Linus
David Woodhouse· Jul 9, 2005, 23:38 UTC · re: Linus Torvalds · lore

Re: What broke snapshots now?

On Sat, 2005-07-09 at 09:15 -0700, Linus Torvalds wrote:
> Yes, looks that way. Except it's not "git on master.kernel.org", it's "git 
> in your home directory", I suspect. I expressly held off packing the 
> kernel repo until git had been updated on kernel.org.

Doh. I thought I'd already done that, but in fact that was for the scripts which feed the mailing list, while the snapshot script kept using my copy. I've moved it out of the way now; sorry for the noise.

-- 
dwmw2
David Woodhouse· Jul 10, 2005, 14:39 UTC · re: David Woodhouse · lore

Re: What broke snapshots now?

On Sun, 2005-07-10 at 00:38 +0100, David Woodhouse wrote:
> Doh. I thought I'd already done that, but in fact that was for the
> scripts which feed the mailing list, while the snapshot script kept
> using my copy.

Ok, the snapshot script starts working again if I change a few environment variables to match what the tools now expect.

Now the mailing list feed isn't happy though -- it stopped being able to pull from your tree at around 0600 UTC (which I think is then the last DRM fix was added). I got this when trying to update...

Tree change: 0109fd37046de64e8459f8c4f4706df9ac7cc82c:f179bc77d09b9087bfc559d0368bba350342ac76 error: cannot read sha1_file for ce68a60e5c503aaef0a98f8d754effb6c7d9ee99 fatal: unable to read destination tree (ce68a60e5c503aaef0a98f8d754effb6c7d9ee99)

Applying changes...
Fast-forwarding 0109fd37046de64e8459f8c4f4706df9ac7cc82c -> f179bc77d09b9087bfc559d0368bba350342ac76
        on top of 0109fd37046de64e8459f8c4f4706df9ac7cc82c...
error: cannot read sha1_file for ce68a60e5c503aaef0a98f8d754effb6c7d9ee99
fatal: failed to unpack tree object f179bc77d09b9087bfc559d0368bba350342ac76

Since it's just a fast-forward, I just copied the 'origin' tag into the 'master' to move it forward. But it's still not happy:

hera /home/dwmw2/git/mail-2.6 $ cg-diff -r 0109fd37046de64e8459f8c4f4706df9ac7cc82c:f179bc77d09b9087bfc559d0368bba350342ac76 error: cannot read sha1_file for ce68a60e5c503aaef0a98f8d754effb6c7d9ee99 fatal: unable to read destination tree (ce68a60e5c503aaef0a98f8d754effb6c7d9ee99)

-- 
dwmw2
Linus Torvalds· Jul 10, 2005, 17:08 UTC · re: David Woodhouse · lore

Re: What broke snapshots now?

On Sun, 10 Jul 2005, David Woodhouse wrote:
> 
> Now the mailing list feed isn't happy though -- it stopped being able to
> pull from your tree at around 0600 UTC (which I think is then the last
> DRM fix was added). I got this when trying to update...

Which script is this? I'm looking at your scripts, but "cg-feedmaillist.sh" is unreadable for me, so I can't see all of it.

Anyway, it's possible that this is a temporary problem: one of the issues is that since you seem to be using the "rsync:" protocol for updating things, what happens is that if the mirroring is off a bit, you may have gotten a new head, but not all the objects. Then you'd get exactly this.

I don't know how to solve that, really. It's fundamental to any mirroring that doesn't guarantee some kind of ordering, ie the dumb kind.

All the "native git" pack-file synchronization protocols should be strictly ordered, ie they will unpack and write the new objects _before_ they write the new refs that point to them..

But you can just imagine what happens if the mirror script is busy
mirroring at the same time, and has already mirrored the object
directories before we wrote all of them, but then gets to the refs/
directory by the time we have written the new refs (and all the new object
- but it didn't notice that because it had already mirrored the object
dirs).

See what I'm saying? The thing fixes itself after a while when you try again and the next round of mirroring has gotten the objects it missed the last time, but there fundamentally isn't anything to be done about the fact that the mirroring wasn't "atomic". Only a git-aware mirror could do an atomic snapshot.

I'm actually surprised that people haven't complained more about this than they have. I know it's happened a couple of times: it bit Dave Jones' scripts once, and it's caused webgit to sometimes report "permission denied" problems (or whatever) and then showing empty logs.

The fact that my upload scripts _do_ upload in the right order, together with the fact that rsync normally syncs alphabetically (ie "objects" before "refs") means that we _normally_ don't see this, but it's occasionally unavoidable when I happen to do a new push while a previous mirror sweep is going on.

Of course, in this case it may also be just some cogito thing that doesn't like pack-files. The fact that it complains about an object that is not actually either of the "heads" of the diff makes me suspect it's really a missing object, though, which in turn makes me suspect the above mirror mis-feature.

			Linus
David Woodhouse· Jul 10, 2005, 17:22 UTC · re: Linus Torvalds · lore

Re: What broke snapshots now?

On Sun, 2005-07-10 at 10:08 -0700, Linus Torvalds wrote:
> Which script is this? I'm looking at your scripts, but
> "cg-feedmaillist.sh" is unreadable for me, so I can't see all of it.

Hm. Dunno why that happened -- it's readable now, and also at http://david.woodhou.se/cg-feedmaillist.sh

> Anyway, it's possible that this is a temporary problem: one of the issues 
> is that since you seem to be using the "rsync:" protocol for updating 
> things, what happens is that if the mirroring is off a bit, you may have 
> gotten a new head, but not all the objects. Then you'd get exactly this.

It's done locally on hera though -- so the mirroring shouldn't be a problem. IIRC the reason it uses rsync is because I wasn't getting tags when it was using whatever other method was the default for a local 'parent repository'.

That was actually more relevant for the snapshots than the mailing list feed, though -- so even if it isn't fixed now, I could live without tags.

More usefully though, if ordering really isn't a problem on your repository then I should probably rewrite the script to work directly from that instead of from a copy.

-- 
dwmw2
Linus Torvalds· Jul 10, 2005, 17:31 UTC · re: David Woodhouse · lore

Re: What broke snapshots now?

On Sun, 10 Jul 2005, David Woodhouse wrote:
Show 8 quoted lines
> 
> > Anyway, it's possible that this is a temporary problem: one of the issues 
> > is that since you seem to be using the "rsync:" protocol for updating 
> > things, what happens is that if the mirroring is off a bit, you may have 
> > gotten a new head, but not all the objects. Then you'd get exactly this.
> 
> It's done locally on hera though -- so the mirroring shouldn't be a
> problem.
No it's not, as far as I can tell:
	torvalds@hera:/home/dwmw2/git/mail-2.6(0)$ cat .git/branches/origin
	rsync://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

so your scripts will go out to rsync with www.kernel.org to get the data, when you use "cg-update origin".

> More usefully though, if ordering really isn't a problem on your
> repository then I should probably rewrite the script to work directly
> from that instead of from a copy.

It should not be a problem, but you obviously have to be careful about ordering, ie you should fetch the references first, and act on them later. 

		Linus
David Woodhouse· Jul 11, 2005, 08:09 UTC · re: Linus Torvalds · lore

Re: What broke snapshots now?

On Sun, 2005-07-10 at 10:31 -0700, Linus Torvalds wrote:
Show 7 quoted lines
> No it's not, as far as I can tell:
> 
>         torvalds@hera:/home/dwmw2/git/mail-2.6(0)$ cat .git/branches/origin
>         rsync://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> so your scripts will go out to rsync with www.kernel.org to get the data, 
> when you use "cg-update origin".

Hm, OK. So I have absolutely no recollection of what my own scripts are actually doing. I could have sworn I made sure it was local. If it was using that URL for the master I might as well have run it elsewhere...

It does seem to be working again now. I'll probably rewrite it next time it misbehaves.

> 
-- 
dwmw2

← back to recent threads