{"thread":{"id":"30261","subject":"git clone fails intermittently if source repo is updated","startedAt":"2012-04-17T12:54:40Z","lastAt":"2012-04-18T08:37:04Z","messageCount":2,"participants":["Per Cederqvist"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"189519","messageId":"87vckyldrj.fsf@opera.com","threadId":"30261","inReplyTo":null,"subject":"git clone fails intermittently if source repo is updated","fromName":"Per Cederqvist","fromEmail":"cederp@opera.com","sentAt":"2012-04-17T12:54:40Z","receivedAt":"2012-04-17T12:54:40Z","isPatch":false,"sender":{"key":"cederp@opera.com","avatar":"https://gravatar.com/avatar/008fe28caedea0ba34042cd5157705267934f4ba7a9b29fb248ae18ec5b3055e?d=mp&s=160"},"body":"Running \"git clone master.git copy\" sometimes fails if changes are being\npushed to master.git.  Likewise, running \"git clone mirror.git copy\"\nsometimes fails if the mirror is currently fetching changes from\nmaster.git.\n\nIn git 1.7.9 I sometimes get one of two fatal error messages.  In one of\nthe cases, git still exits with exit code 0, but the working directory\nof \"copy\" is not updated with the proper files.  (See below for the\nexact error messages.)\n\nIn git 1.7.10 the mirror sometimes ends up in \"detached HEAD\" state.\nSometimes it prints a warning message.  (See below for the messages.)\n\nThe following shell scripts can reproduce the problem.  I've been using\na 4-core CPU on Ubuntu 11.04.  I've compiled Git from source.\n\nThe first script, test-git-setup.sh, creates the directory \"test-git\"\nand three git repositories in it: master.git, mirror.git, and committer:\n\n------ cut here for test-git-setup.sh ------\n#!/bin/sh\nmkdir test-git || exit 1\ncd test-git\nmkdir master.git\n(cd master.git && git init --bare)\ngit clone master.git committer\n(cd committer &&\n    touch foo &&\n    git add foo &&\n    git commit -m\"Initial commit\" &&\n    git push origin master)\ngit clone --mirror master.git mirror.git\n------ cut here ------\n\nThe second script continuously modifies the file \"foo\" in the committer\nrepository, commits it, and pushes it to master.git:\n\n------ cut here for test-git-commit.sh ------\n#!/bin/sh\nset -e\ncd test-git/committer\ni=0\nwhile :\ndo\n    echo $i > foo\n    git add foo\n    git commit -m\"Updated foo\"\n    git push\n    i=`expr $i + 1`\ndone\n------ cut here ------\n\nThe third script repeatedly clones a repository, and ensures that the\n\"git clone\" command does exit 0, prints no unexpected output, and\ncreates a sane clone (the file \"foo\" must exist).  You must give it the\nargument \"master\" or \"mirror\" to tell it which repository to clone.\n\n------ cut here for test-git-reclone.sh ------\n#!/bin/sh\n\nwhat=$1\n\ncase \"$what\" in\n    master);;\n    mirror);;\n    *) echo need to specify master or mirror >&2\n       exit 1;;\nesac\n\ngood=0\ncd test-git || exit 1\nrm -rf $what-clone\nwhile :\ndo\n    git clone $what.git $what-clone >$what.out 2>$what.err || exit 1\n    if [ -s $what.err ]\n    then\n\techo unexpected stderr from git clone:\n\tcat $what.err\n\texit 1\n    fi\n    if [ \"`cat $what.out`\" != \"Cloning into '$what-clone'...\ndone.\" ]\n    then\n\techo unexpected stdout from git clone:\n\tcat $what.out\n\texit 1\n    fi\n    if ! [ -f $what-clone/foo ]\n    then\n\techo failed to create $what-clone/foo\n\texit 1\n    fi\n    rm -rf $what-clone\n    good=`expr $good + 1`\n    echo \"$good good clones\"\ndone\n------ cut here ------\n\nThe final script repeatedly runs \"git fetch\" in mirror.git, to update it\nfrom master.git.\n\n------ cut here for test-git-mirror.sh ------\n#!/bin/sh\ncd test-git/mirror.git\nwhile :\ndo\n    echo running fetch...\n    git fetch || exit 1\n    echo done.\ndone\n------ cut here ------\n\nTo reproduce, run this command:\n\n    ./test-git-setup.sh\n\nWhen it has completed, run the following commands at once (preferably\nin four separate windows so you can see what is going on).  Start them\nin the indicated order.\n\n    ./test-git-reclone.sh master\n    ./test-git-reclone.sh mirror\n    ./test-git-mirror.sh\n    ./test-git-commit.sh\n\nThe reclone commands will print lines such as these:\n\n    1 good clones\n    2 good clones\n    3 good clones\n    4 good clones\n\nLet them run to at least 200 good clones before you decide that you\ncannot reproduce the problem.\n\nUsing Git 1.7.10 I get the following error after a while:\n\n> unexpected stderr from git clone:\n> Note: checking out 'c210776284dc8bc1c3f6eecc316b539c9aea85c9'.\n>\n> You are in 'detached HEAD' state. You can look around, make experimental\n> changes and commit them, and you can discard any commits you make in this\n> state without impacting any branches by performing another checkout.\n>\n> If you want to create a new branch to retain commits you create, you may\n> do so (now or later) by using -b with the checkout command again. Example:\n>\n>   git checkout -b new_branch_name\n>\n\nWhen this happens, I can recover by running \"git checkout master\" in the\nclone, but before I do so \"git status\" reports:\n\n> # Not currently on any branch.\n> nothing to commit (working directory clean)\n\nand \"git branch -a\" reports:\n\n> * (no branch)\n>   remotes/origin/master\n\nSometimes I get a different error:\n\n> unexpected stderr from git clone:\n> warning: failed to stat /home/cederp/test-git/mirror.git/objects/f6/tmp_obj_TZjRRd\n>\n\nor\n\n> unexpected stderr from git clone:\n> warning: failed to stat /home/cederp/test-git/master.git/objects/59/tmp_obj_SZtkWc\n>\n\nUsing Git 1.7.9 I sometimes get this error when running\n\"./test-git-reclone.sh mirror\":\n\n> Cloning into 'mirror-clone'...\n> done.\n> error: refs/remotes/origin/master does not point to a valid object!\n> error: Trying to write ref refs/heads/master with nonexistent object 5d049be4cd93c230a88503d8f9b4057587c76b2d\n> fatal: Cannot update the ref 'HEAD'.\n\nSometimes I get this message:\n\n> Cloning into 'mirror-clone'...\n> done.\n> error: unable to find 1338993ab85d24f451be1cd6565c4685c24fcc15\n> error: unable to read sha1 file of foo (1338993ab85d24f451be1cd6565c4685c24fcc15)\n\nIn this case, \"git clone\" does exit(0), but the file \"foo\" does not\nexist in the created mirror.\n\n    /ceder\n"},{"id":"189609","messageId":"4F8E7D30.2010906@opera.com","threadId":"30261","inReplyTo":"87vckyldrj.fsf@opera.com","subject":"Re: git clone fails intermittently if source repo is updated","fromName":"Per Cederqvist","fromEmail":"cederp@opera.com","sentAt":"2012-04-18T08:37:04Z","receivedAt":"2012-04-18T08:37:04Z","isPatch":false,"sender":{"key":"cederp@opera.com","avatar":"https://gravatar.com/avatar/008fe28caedea0ba34042cd5157705267934f4ba7a9b29fb248ae18ec5b3055e?d=mp&s=160"},"body":"On 04/17/2012 02:54 PM, Per Cederqvist wrote:\n\n> Running \"git clone master.git copy\" sometimes fails if changes are being\n> pushed to master.git.  Likewise, running \"git clone mirror.git copy\"\n> sometimes fails if the mirror is currently fetching changes from\n> master.git.\n>\n> In git 1.7.9 I sometimes get one of two fatal error messages.  In one of\n> the cases, git still exits with exit code 0, but the working directory\n> of \"copy\" is not updated with the proper files.  (See below for the\n> exact error messages.)\n>\n> In git 1.7.10 the mirror sometimes ends up in \"detached HEAD\" state.\n> Sometimes it prints a warning message.  (See below for the messages.)\n>\n> The following shell scripts can reproduce the problem.  I've been using\n> a 4-core CPU on Ubuntu 11.04.  I've compiled Git from source.\n>\n> The first script, test-git-setup.sh, creates the directory \"test-git\"\n> and three git repositories in it: master.git, mirror.git, and committer:\n>\n > [...four shellscripts omitted...]\n\nFor your convenience, I have now set up a Git repository where you can\nget slightly updated versions of the above scripts.  The Gitorious web\npage is here:\n\nhttp://git.lysator.liu.se/ceder/git-clone-fail-demo\n\nYou can clone it like this:\n\ngit clone git://git.lysator.liu.se/ceder/git-clone-fail-demo.git\n\nI have been able to reproduce the problem on Git 1.6.0 as well, so\nthis is apparently not a recent regression.\n\nThe problem can also be reproduces when you clone, push and fetch via\nssh, but it happens a lot less frequently then.  See the README file in\nthe git-clone-fail-demo repository for info on how to reproduce it.\n\n> [...]\n>\n>      /ceder\n\n     /ceder\n"}]}