{"thread":{"id":"26653","subject":"git-pack-objects dying with errors due to possible RHEL kernel bug","startedAt":"2011-03-04T11:08:56Z","lastAt":"2011-04-20T21:39:04Z","messageCount":2,"participants":["Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"162763","messageId":"AANLkTimZ08_fJKgwqtiKBbd0pSJ3EFSnJ6jem=fPqRWw@mail.gmail.com","threadId":"26653","inReplyTo":null,"subject":"git-pack-objects dying with errors due to possible RHEL kernel bug","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2011-03-04T11:08:56Z","receivedAt":"2011-03-04T11:08:56Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This is a summary of an issue I've been looking at with a very large\ncentralized Git repository. It's a repository that gets approximately\n100 commits per day, almost all to its master branch.\n\nI think I've found why the issue I'm describing happens (not confirmed\nyet), I mainly wanted to write something to the list to have a record\nof this in case anyone runs into it in the future.\n\nLast week we upgraded form Git 1.6.5 to 1.7.2.1 on the server housing\nour repository, and started getting errors like these from developers\nrunning variants of git-fetch:\n\n    $ git pull --rebase\n    remote: Counting objects: 2, done.\n    remote: Compressing objects: 100% (2/2), done.\n    remote: Total 2 (delta 0), reused 0 (delta 0)\n    remote: aborting due to possible repository corruption on the remote side.\n    error: waitpid for pack-objects failed: No child processes\n    error: git upload-pack: git-pack-objects died with error.\n    fatal: git upload-pack: aborting due to possible repository\ncorruption on the remote side.\n    Unpacking objects: 100% (2/2), done.\n    fatal: error in sideband demultiplexer\n\nThat error is from\nhttps://github.com/git/git/commit/b1c71b72815cb82a8bad14020a047320b88a04eb\nby Junio from 2006, we're refusing to send an incomplete pack file on\nfailure.\n\nWe've also been getting this error from git-fetch directly (from a\nwrapper script):\n\n    # INFO : Checking working directory\n    # ERROR: failed to git fetch --tags from 'origin' errorcode: 128\n    # ERROR: git fetch --tags origin\n    # ERROR: error: waitpid for pack-objects failed: No child processes\n    # ERROR: error: git upload-pack: git-pack-objects died with error.\n    # ERROR: fatal: git upload-pack: aborting due to possible\nrepository corruption on the remote side.\n    # ERROR: remote: aborting due to possible repository corruption on\nthe remote side.\n    # ERROR: fatal: error in sideband demultiplexer\n\nAnd from git-remote-update(1):\n\n    $ git remote update\n    Fetching origin\n    remote: Counting objects: 9, done.\n    remote: Compressing objects: 100% (5/5), done.\n    remote: Total 5 (delta 4), reused 0 (delta 0)\n    error: waitpid for pack-objects failed: No child processes\n    error: git upload-pack: git-pack-objects died with error.\n    fatal: git upload-pack: aborting due to possible repository\ncorruption on the remote side.\n    remote: aborting due to possible repository corruption on the remote side.\n    Unpacking objects: 100% (5/5), done.\n    fatal: error in sideband demultiplexer\n    error: Could not fetch origin\n\nAll of these except maybe the first one (wasn't able to contact the\ndev in question) come from Git 1.7.2.1 clients talking to the 1.7.2.1\nserver.\n\nAnyway, I think this issue is caused by this RHEL bug:\nhttps://bugzilla.redhat.com/show_bug.cgi?id=166669 ([RHEL3 U5]\nwaitpid() returns unexpected ECHILD) which was fixed in this RHEL\nupdate: http://rhn.redhat.com/errata/RHSA-2006-0144.html\n\nThis is our Git server:\n\n    $ cat /etc/redhat-release && uname -r\n    CentOS release 4.1 (Final)\n    2.6.9-11.ELsmp\n\nAnd if I run:\n\n    wget https://bugzilla.redhat.com/attachment.cgi?id=118759 -O killipf.c &&\n    gcc -O2 -o killipf killipf.c -lpthread &&\n    PASS=0; while ./killipf; do let PASS=++PASS; echo $PASS; done\n\nIt'll die within a minute with a message like this:\n\n    PASS : received expected signal 9\n    14605\n\n    child pid:2563\n    waitpid failed!: No child processes\n\nIt does *not* die on these machines:\n\n    $ cat /etc/redhat-release && uname -r\n    CentOS release 4.6 (Final)\n    2.6.9-67.0.7.ELsmp\n\n    $ cat /etc/redhat-release && uname -r\n    CentOS release 5.5 (Final)\n    2.6.18-194.el5PAE\n\nOr on my personal Debian box:\n\n    $ cat /etc/debian_version && uname -r\n    wheezy/sid\n    2.6.32-5-amd64\n\nI haven't been able to trigger this issue with Git itself. I tried\nputting a copy of the repository in /tmp, then one on client running\nin a while loop:\n\n    while true; do\n        head -n 10 /dev/urandom >a_file &&\n        git commit -m\"more crap\" a_file &&\n        git push\n    done\n\nAnd on another client running:\n\n    while true; do\n        git pull\n    done\n\nAnd I never got this waitpid error message, I might have just been\nunlucky though, or perhaps it wasn't triggered in that case for some\nreason.\n\nGiven this information we're going to upgrade CentOS on the relevant\nmachine, I'll follow up on the list in a couple of weeks indicating\nwhether or not that worked. We have enough users that if I ask people\nto tell me if we get this error and I don't hear anything for two\nweeks I can safely assume it went away.\n\nWhat we might want to do in Git is to work around this broken waitpid\nbehavior (if that's indeed the issue). I haven't dug into what the\nRHEL kernel patch is solving, so I don't know if we can inexpensively\ndetect this when this is happening and warn users about it.\n\nThen again it would be a lot of work to work around a specific kernel\nbug. What I *mainly* wanted to do was to insert some note of this into\nthe Git mailing list archive. Which I've now done.\n"},{"id":"166161","messageId":"BANLkTimhNQrqhOq36-9wm9Zy=nL7Epk2Pw@mail.gmail.com","threadId":"26653","inReplyTo":"AANLkTimZ08_fJKgwqtiKBbd0pSJ3EFSnJ6jem=fPqRWw@mail.gmail.com","subject":"Re: git-pack-objects dying with errors due to possible RHEL kernel bug","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2011-04-20T21:39:04Z","receivedAt":"2011-04-20T21:39:04Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Mar 4, 2011 at 12:08, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> This is a summary of an issue I've been looking at with a very large\n> centralized Git repository. It's a repository that gets approximately\n> 100 commits per day, almost all to its master branch.\n>\n> I think I've found why the issue I'm describing happens (not confirmed\n> yet), I mainly wanted to write something to the list to have a record\n> of this in case anyone runs into it in the future.\n>\n> Last week we upgraded form Git 1.6.5 to 1.7.2.1 on the server housing\n> our repository, and started getting errors like these from developers\n> running variants of git-fetch:\n>\n>    $ git pull --rebase\n>    remote: Counting objects: 2, done.\n>    remote: Compressing objects: 100% (2/2), done.\n>    remote: Total 2 (delta 0), reused 0 (delta 0)\n>    remote: aborting due to possible repository corruption on the remote side.\n>    error: waitpid for pack-objects failed: No child processes\n>    error: git upload-pack: git-pack-objects died with error.\n>    fatal: git upload-pack: aborting due to possible repository\n> corruption on the remote side.\n>    Unpacking objects: 100% (2/2), done.\n>    fatal: error in sideband demultiplexer\n>\n> That error is from\n> https://github.com/git/git/commit/b1c71b72815cb82a8bad14020a047320b88a04eb\n> by Junio from 2006, we're refusing to send an incomplete pack file on\n> failure.\n>\n> We've also been getting this error from git-fetch directly (from a\n> wrapper script):\n>\n>    # INFO : Checking working directory\n>    # ERROR: failed to git fetch --tags from 'origin' errorcode: 128\n>    # ERROR: git fetch --tags origin\n>    # ERROR: error: waitpid for pack-objects failed: No child processes\n>    # ERROR: error: git upload-pack: git-pack-objects died with error.\n>    # ERROR: fatal: git upload-pack: aborting due to possible\n> repository corruption on the remote side.\n>    # ERROR: remote: aborting due to possible repository corruption on\n> the remote side.\n>    # ERROR: fatal: error in sideband demultiplexer\n>\n> And from git-remote-update(1):\n>\n>    $ git remote update\n>    Fetching origin\n>    remote: Counting objects: 9, done.\n>    remote: Compressing objects: 100% (5/5), done.\n>    remote: Total 5 (delta 4), reused 0 (delta 0)\n>    error: waitpid for pack-objects failed: No child processes\n>    error: git upload-pack: git-pack-objects died with error.\n>    fatal: git upload-pack: aborting due to possible repository\n> corruption on the remote side.\n>    remote: aborting due to possible repository corruption on the remote side.\n>    Unpacking objects: 100% (5/5), done.\n>    fatal: error in sideband demultiplexer\n>    error: Could not fetch origin\n>\n> All of these except maybe the first one (wasn't able to contact the\n> dev in question) come from Git 1.7.2.1 clients talking to the 1.7.2.1\n> server.\n>\n> Anyway, I think this issue is caused by this RHEL bug:\n> https://bugzilla.redhat.com/show_bug.cgi?id=166669 ([RHEL3 U5]\n> waitpid() returns unexpected ECHILD) which was fixed in this RHEL\n> update: http://rhn.redhat.com/errata/RHSA-2006-0144.html\n>\n> This is our Git server:\n>\n>    $ cat /etc/redhat-release && uname -r\n>    CentOS release 4.1 (Final)\n>    2.6.9-11.ELsmp\n>\n> And if I run:\n>\n>    wget https://bugzilla.redhat.com/attachment.cgi?id=118759 -O killipf.c &&\n>    gcc -O2 -o killipf killipf.c -lpthread &&\n>    PASS=0; while ./killipf; do let PASS=++PASS; echo $PASS; done\n>\n> It'll die within a minute with a message like this:\n>\n>    PASS : received expected signal 9\n>    14605\n>\n>    child pid:2563\n>    waitpid failed!: No child processes\n>\n> It does *not* die on these machines:\n>\n>    $ cat /etc/redhat-release && uname -r\n>    CentOS release 4.6 (Final)\n>    2.6.9-67.0.7.ELsmp\n>\n>    $ cat /etc/redhat-release && uname -r\n>    CentOS release 5.5 (Final)\n>    2.6.18-194.el5PAE\n>\n> Or on my personal Debian box:\n>\n>    $ cat /etc/debian_version && uname -r\n>    wheezy/sid\n>    2.6.32-5-amd64\n>\n> I haven't been able to trigger this issue with Git itself. I tried\n> putting a copy of the repository in /tmp, then one on client running\n> in a while loop:\n>\n>    while true; do\n>        head -n 10 /dev/urandom >a_file &&\n>        git commit -m\"more crap\" a_file &&\n>        git push\n>    done\n>\n> And on another client running:\n>\n>    while true; do\n>        git pull\n>    done\n>\n> And I never got this waitpid error message, I might have just been\n> unlucky though, or perhaps it wasn't triggered in that case for some\n> reason.\n>\n> Given this information we're going to upgrade CentOS on the relevant\n> machine, I'll follow up on the list in a couple of weeks indicating\n> whether or not that worked. We have enough users that if I ask people\n> to tell me if we get this error and I don't hear anything for two\n> weeks I can safely assume it went away.\n>\n> What we might want to do in Git is to work around this broken waitpid\n> behavior (if that's indeed the issue). I haven't dug into what the\n> RHEL kernel patch is solving, so I don't know if we can inexpensively\n> detect this when this is happening and warn users about it.\n>\n> Then again it would be a lot of work to work around a specific kernel\n> bug. What I *mainly* wanted to do was to insert some note of this into\n> the Git mailing list archive. Which I've now done.\n\nAn update: after upgrading CentOS past a kernel with this bug we\nstopped having these errors. We used to have many of them every day,\nbut haven't had one in the week since the server was upgraded.\n"}]}