Massive repository corruptions

Hi folks,

I've just reorganized several repositories (eg. splitted off a large repo into several small ones), and then I had massive corruptions (broken pack files) in the new repos (after they already had been clean).

Maybe it has something to do with a cronjob which frequently GC's all the repos, and it could get even worse if the fs sometimes goes full within this process.

Could multiple GCs running on the same repo cause this ?

cu

On Mon, Jul 12, 2010 at 9:56 PM, Enrico Weigelt <weigelt@metux.de> wrote:

Multiple simultaneous gc's shouldn't be a problem - git locks things as it needs them. Plus, git only removes objects after it has safely created a new packfile that contains them. Maybe a filesystem filling up could cause a problem, but git should be detecting that if it happens (maybe there's a bug that causes it to not notice, though).

You could experience corruption if your computer crashed before everything was synced to disk.

Do you know which packfiles are corrupted? Does 'git index-pack' on the files reveal anything?

Be sure to make a backup copy of your corrupted repositories before doing any experiments, or you might accidentally fix the problem and make it harder to trace.

Good luck.

Avery

* Avery Pennarun <apenwarr@gmail.com> wrote:

Okay.

No machine crash, and no sign of filesystem or disk problems (according to kernel log).

git@blackwidow ~/metux/work.git/pack $ git index-pack pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack error: inflate: data stream error (incorrect data check) fatal: pack has bad object at offset 37075832: inflate returned -3

(that's essentially the same git-gc says)

git@blackwidow ~/metux/work.git/pack $ git unpack-objects -r < pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack error: inflate: data stream error (incorrect data check) error: inflate returned -3

error: inflate: data stream error (incorrect data check) error: inflate returned -3

Unpacking objects: 100% (1223/1223), done. fatal: final sha1 did not match

cu

* Enrico Weigelt <weigelt@metux.de> wrote:

<snip>

What's strange: 

when copying pack files from another machine to this box and run git index-pack there, it fails with the same error.

also: pushing into a new (bare) repo sometimes fails with inflate errors, sometimes succeeds but leaves an broken packfile.

cu

* Enrico Weigelt <weigelt@metux.de> wrote:

Interesting: if I limit the pack size on the local repository,
and manually copy them over via scp, git index-pack runs fine
on them.

Subsequent push doesnt seem to recognize the already transferred packs and still sends the big one which gets broken, but running git-gc multiple times seems to clean up the mess.

Is there any way for limiting the pack size on push ? (pack.packSizeLimit only affects git-repack, not remote transfers).

cu

On 13 July 2010 06:31, Enrico Weigelt <weigelt@metux.de> wrote:

The pack files you copied over from another machine, were they sane (i.e. non-corrupt)? If so, that perhaps smells like your hard drive could be on its last legs...

On Tue, Jul 13, 2010 at 1:03 AM, Enrico Weigelt <weigelt@metux.de> wrote:

What's the size of that .pack file?

Avery

* Avery Pennarun <apenwarr@gmail.com> wrote:

Somewhat over 300MB. 

Lowering the packfile size seemed to help. (but I still only can do that for git-repack, not remote transfers)

cu

On Tue, Jul 13, 2010 at 6:22 AM, Enrico Weigelt <weigelt@metux.de> wrote:

If you got corruption at offset 37,075,832 (about 37 megs) and the pack is over 300 megs, then the file itself is corrupted right in the middle (not truncated) and this couldn't have been caused by disk full errors. Either you have memory corruption problems, or disk corruption problems, or filesystem corruption problems. You'd better watch out.

Forcing the packfile size to be smaller probably just changes your memory access patterns and moves your errors around. But it doesn't sound like a git bug at this point.

Avery

* Avery Pennarun <apenwarr@gmail.com> wrote:

hmm, I have no signs of any hw corruption, but I had a patched version of zlib installed. Maybe some of my patches broke it, so some strange overflow or sth like that caused that trouble.

Meanwhile, after reinstalling (unpatched) zlib and recloning the broken repos, everything seems fine again. Maybe some of you would like to have a look at my zlib patches ;-o

cu

* Enrico Weigelt <weigelt@metux.de> wrote:

This only seemed to help for a while. Again have trouble w/ broken repos. But the strange thing: seems to affect only large ones. For example, could got clone and repeatedly gc --aggressive the git source w/ trouble.

If it *is* any hw problem (which isnt that unplausible since that machine is the only one making trouble now), how can I detect it ? Shouldnt broken memory or disk raise some kernel log message ?

cu

* Jussi Sirpoma <jussi.sirpoma@gmail.com> wrote:

hmm, you know some way to do a memory-stresstest w/o rebooting ?

cu

* Jussi Sirpoma <jussi.sirpoma@sks.fi> wrote:

okay, I'll have a try :)

BTW: I'm currently recreating one of the broken repos (mozilla,
which might be large enough for stresstest ;-)) by adding and
fetching the remotes step by step. Reposity size now about 250M
(reduced the packsize to 32M, since this already helped on some
other repos) - yet no breaks occoured.

Let's see where it goes ...

cu