git-pack-objects dying with errors due to possible RHEL kernel bug

This is a summary of an issue I've been looking at with a very large centralized Git repository. It's a repository that gets approximately 100 commits per day, almost all to its master branch.

I think I've found why the issue I'm describing happens (not confirmed yet), I mainly wanted to write something to the list to have a record of this in case anyone runs into it in the future.

Last week we upgraded form Git 1.6.5 to 1.7.2.1 on the server housing our repository, and started getting errors like these from developers running variants of git-fetch:

    $ git pull --rebase
    remote: Counting objects: 2, done.
    remote: Compressing objects: 100% (2/2), done.
    remote: Total 2 (delta 0), reused 0 (delta 0)
    remote: aborting due to possible repository corruption on the remote side.
    error: waitpid for pack-objects failed: No child processes
    error: git upload-pack: git-pack-objects died with error.
    fatal: git upload-pack: aborting due to possible repository
corruption on the remote side.
    Unpacking objects: 100% (2/2), done.
    fatal: error in sideband demultiplexer

That error is from https://github.com/git/git/commit/b1c71b72815cb82a8bad14020a047320b88a04eb by Junio from 2006, we're refusing to send an incomplete pack file on failure.

We've also been getting this error from git-fetch directly (from a wrapper script):

    # INFO : Checking working directory
    # ERROR: failed to git fetch --tags from 'origin' errorcode: 128
    # ERROR: git fetch --tags origin
    # ERROR: error: waitpid for pack-objects failed: No child processes
    # ERROR: error: git upload-pack: git-pack-objects died with error.
    # ERROR: fatal: git upload-pack: aborting due to possible
repository corruption on the remote side.
    # ERROR: remote: aborting due to possible repository corruption on
the remote side.
    # ERROR: fatal: error in sideband demultiplexer

And from git-remote-update(1):

    $ git remote update
    Fetching origin
    remote: Counting objects: 9, done.
    remote: Compressing objects: 100% (5/5), done.
    remote: Total 5 (delta 4), reused 0 (delta 0)
    error: waitpid for pack-objects failed: No child processes
    error: git upload-pack: git-pack-objects died with error.
    fatal: git upload-pack: aborting due to possible repository
corruption on the remote side.
    remote: aborting due to possible repository corruption on the remote side.
    Unpacking objects: 100% (5/5), done.
    fatal: error in sideband demultiplexer
    error: Could not fetch origin

All of these except maybe the first one (wasn't able to contact the dev in question) come from Git 1.7.2.1 clients talking to the 1.7.2.1 server.

Anyway, I think this issue is caused by this RHEL bug: https://bugzilla.redhat.com/show_bug.cgi?id=166669 ([RHEL3 U5] waitpid() returns unexpected ECHILD) which was fixed in this RHEL update: http://rhn.redhat.com/errata/RHSA-2006-0144.html

This is our Git server:

    $ cat /etc/redhat-release && uname -r
    CentOS release 4.1 (Final)
    2.6.9-11.ELsmp

And if I run:

    wget https://bugzilla.redhat.com/attachment.cgi?id=118759 -O killipf.c &&
    gcc -O2 -o killipf killipf.c -lpthread &&
    PASS=0; while ./killipf; do let PASS=++PASS; echo $PASS; done

It'll die within a minute with a message like this:

    PASS : received expected signal 9
    14605

    child pid:2563
    waitpid failed!: No child processes

It does *not* die on these machines:

    $ cat /etc/redhat-release && uname -r
    CentOS release 4.6 (Final)
    2.6.9-67.0.7.ELsmp

    $ cat /etc/redhat-release && uname -r
    CentOS release 5.5 (Final)
    2.6.18-194.el5PAE

Or on my personal Debian box:

    $ cat /etc/debian_version && uname -r
    wheezy/sid
    2.6.32-5-amd64

I haven't been able to trigger this issue with Git itself. I tried putting a copy of the repository in /tmp, then one on client running in a while loop:

    while true; do
        head -n 10 /dev/urandom >a_file &&
        git commit -m"more crap" a_file &&
        git push
    done

And on another client running:

    while true; do
        git pull
    done

And I never got this waitpid error message, I might have just been unlucky though, or perhaps it wasn't triggered in that case for some reason.

Given this information we're going to upgrade CentOS on the relevant machine, I'll follow up on the list in a couple of weeks indicating whether or not that worked. We have enough users that if I ask people to tell me if we get this error and I don't hear anything for two weeks I can safely assume it went away.

What we might want to do in Git is to work around this broken waitpid behavior (if that's indeed the issue). I haven't dug into what the RHEL kernel patch is solving, so I don't know if we can inexpensively detect this when this is happening and warn users about it.

Then again it would be a lot of work to work around a specific kernel bug. What I *mainly* wanted to do was to insert some note of this into the Git mailing list archive. Which I've now done.

On Fri, Mar 4, 2011 at 12:08, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

An update: after upgrading CentOS past a kernel with this bug we stopped having these errors. We used to have many of them every day, but haven't had one in the week since the server was upgraded.