{"thread":{"id":"30236","subject":"master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","startedAt":"2012-04-14T09:01:12Z","lastAt":"2012-04-23T11:35:05Z","messageCount":26,"participants":["Stefano Lattarini","Clemens Buchacher","Sverre Rabbelier","Pete Wyckoff","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"189254","messageId":"4F893CD8.5020700@gmail.com","threadId":"30236","inReplyTo":null,"subject":"master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-14T09:01:12Z","receivedAt":"2012-04-14T09:01:12Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"The test 't5800-remote-helpers.sh' hangs on my Linux desktop (where I have\ninstalled a custom simple-minded firewall based on iptables). This does\nnot happen on remote systems I use for testing, so I guess the problem\nlies with my machine. Still, having the test to hang is unnecessarily\nharsh and user-unfriendly IMHO, and it would be better to detect the\nsituation and SKIP or FAIL the affected test cases instead.\n\nFor reference, here is the output of the test 't5800-remote-helpers.sh' run\nwith the '-v' option:\n\n-*-*-\n\nInitialized empty Git repository in /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/.git/\nexpecting success:\n git init --bare server/.git &&\n git clone server public &&\n (cd public &&\n echo content >file &&\n git add file &&\n git commit -m one &&\n git push origin master)\n\nInitialized empty Git repository in /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git/\nCloning into 'public'...\nwarning: You appear to have cloned an empty repository.\ndone.\n[master (root-commit) 9a7aa46] one\n Author: A U Thor \n 1 file changed, 1 insertion(+)\n create mode 100644 file\nCounting objects: 3, done.\nUnpacking objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 219 bytes, done.\nTotal 3 (delta 0), reused 0 (delta 0)\nTo /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n * [new branch] master -> master\nok 1 - setup repository\n\nexpecting success:\n git clone \"testgit::${PWD}/server\" localclone &&\n test_cmp public/file localclone/file\n\nCloning into 'localclone'...\nok 2 - cloning from local repo\n\nexpecting success:\n git clone \"testgit::file://${PWD}/server\" clone &&\n test_cmp public/file clone/file\n\nCloning into 'clone'...\nok 3 - cloning from remote repo\n\nexpecting success:\n (cd public &&\n echo content >>file &&\n git commit -a -m two &&\n git push)\n\n[master 0eb255a] two\n Author: A U Thor \n 1 file changed, 1 insertion(+)\nCounting objects: 5, done.\nUnpacking objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 252 bytes, done.\nTotal 3 (delta 0), reused 0 (delta 0)\nTo /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n 9a7aa46..0eb255a master -> master\nok 4 - create new commit on remote\n\nexpecting success:\n (cd localclone && git pull) &&\n test_cmp public/file localclone/file\n\n>From testgit::/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n 9a7aa46..0eb255a master -> origin/master\nUpdating 9a7aa46..0eb255a\nFast-forward\n file | 1 +\n 1 file changed, 1 insertion(+)\nok 5 - pulling from local repo\n\nexpecting success:\n (cd clone && git pull) &&\n test_cmp public/file clone/file\n\n>From testgit::file:///devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n 9a7aa46..0eb255a master -> origin/master\nUpdating 9a7aa46..0eb255a\nFast-forward\n file | 1 +\n 1 file changed, 1 insertion(+)\nok 6 - pulling from remote remote\n\nexpecting success:\n (cd localclone &&\n echo content >>file &&\n git commit -a -m three &&\n git push) &&\n compare_refs localclone HEAD server HEAD\n\n[master 30d4cae] three\n Author: A U Thor \n 1 file changed, 1 insertion(+)\n\n-*-*-\n\nAt this point, the test hangs, and I have to manually kill it with a SIGINT,\nthe result being:\n\n-*-*-\n\nTraceback (most recent call last):\n File \"/devel/stefano/src/git/git-remote-testgit\", line 265, in \n sys.exit(main(sys.argv))\n File \"/devel/stefano/src/git/git-remote-testgit\", line 262, in main\n more = read_one_line(repo)\n File \"/devel/stefano/src/git/git-remote-testgit\", line 227, in read_one_line\n func(repo, cmdline)\n File \"/devel/stefano/src/git/git-remote-testgit\", line 175, in do_export\n changed = repo.importer.do_import(repo.gitdir)\n File \"/home/stefano/src/git/t/../git_remote_helpers/build/lib/git_remote_helpers/git/importer.py\", line 54, in do_import\n check_call(args)\n File \"/home/stefano/src/git/t/../git_remote_helpers/build/lib/git_remote_helpers/util.py\", line 169, in check_call\n retcode = call(*popenargs, **kwargs)\n File \"/home/stefano/src/git/t/../git_remote_helpers/build/lib/git_remote_helpers/util.py\", line 155, in call\n return subprocess.Popen(*popenargs, **kwargs).wait()\n File \"/usr/lib/python2.7/subprocess.py\", line 1291, in wait\n pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)\n File \"/usr/lib/python2.7/subprocess.py\", line 478, in _eintr_retry_call\n return func(*args)\nKeyboardInterrupt\nFATAL: Unexpected exit with code 0\n\n-*-*-\n\nIf I run ps(1) in another terminal before interrupting the hung test, the\nfollowing command sticks out as a possible culprit:\n\ngit-fast-import --quiet --export-marks=/devel/stefano/src/git/t/trash \\\ndirectory.t5800-remote-helpers/localclone/.git/info/fast-import/\\\nc6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks \\\n--import-marks=/devel/stefano/src/git/t/'trash directory.t5800-remote-helpers'/\\\nlocalclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/\\\ngit.marks\n\n-*-*-\n\nOther possibly relevant information:\n\n $ hg --version | head -1\n Mercurial Distributed SCM (version 1.8.3) $ python --version\n\n $ python --version\n Python 2.7.2+\n\n $ /bin/sh --version | head -1\n GNU bash, version 4.1.5(1)-release (i486-pc-linux-gnu)\n\n $ git describe\n v1.7.10-167-gf245c01\n"},{"id":"189284","messageId":"20120414201446.GB29999@ecki","threadId":"30236","inReplyTo":"4F893CD8.5020700@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-14T20:14:48Z","receivedAt":"2012-04-14T20:14:48Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sat, Apr 14, 2012 at 11:01:12AM +0200, Stefano Lattarini wrote:\n>\n> The test 't5800-remote-helpers.sh' hangs on my Linux desktop (where I have\n> installed a custom simple-minded firewall based on iptables). This does\n> not happen on remote systems I use for testing, so I guess the problem\n> lies with my machine.\n\nI don't think this particular test is related to networking. Could you\nplease run the test again with the following change and paste the\noutput? (You cannot set the variables on the command line, because\ntest-lib.sh would unset them.)\n\ndiff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\nindex 1c62001..4cebd63 100755\n--- a/t/t5800-remote-helpers.sh\n+++ b/t/t5800-remote-helpers.sh\n@@ -68,7 +68,7 @@ test_expect_success 'pushing to local repo' '\n \t(cd localclone &&\n \techo content >>file &&\n \tgit commit -a -m three &&\n-\tgit push) &&\n+\tGIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n \tcompare_refs localclone HEAD server HEAD\n '\n \n"},{"id":"189307","messageId":"4F8A0F96.5060408@gmail.com","threadId":"30236","inReplyTo":"20120414201446.GB29999@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T00:00:22Z","receivedAt":"2012-04-15T00:00:22Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/14/2012 10:14 PM, Clemens Buchacher wrote:\n> On Sat, Apr 14, 2012 at 11:01:12AM +0200, Stefano Lattarini wrote:\n>>\n>> The test 't5800-remote-helpers.sh' hangs on my Linux desktop (where I have\n>> installed a custom simple-minded firewall based on iptables). This does\n>> not happen on remote systems I use for testing, so I guess the problem\n>> lies with my machine.\n> \n> I don't think this particular test is related to networking. Could you\n> please run the test again with the following change and paste the\n> output? (You cannot set the variables on the command line, because\n> test-lib.sh would unset them.)\n> \n> diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\n> index 1c62001..4cebd63 100755\n> --- a/t/t5800-remote-helpers.sh\n> +++ b/t/t5800-remote-helpers.sh\n> @@ -68,7 +68,7 @@ test_expect_success 'pushing to local repo' '\n> \t(cd localclone &&\n> \techo content >>file &&\n> \tgit commit -a -m three &&\n> -\tgit push) &&\n> +\tGIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n> \tcompare_refs localclone HEAD server HEAD\n> '\n> \nHere it goes:\n\n-*-*-\n\n$ ./t5800-remote-helpers.sh -v\n\nInitialized empty Git repository in /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/.git/\nexpecting success:\n git init --bare server/.git &&\n git clone server public &&\n (cd public &&\n echo content >file &&\n git add file &&\n git commit -m one &&\n git push origin master)\n\nInitialized empty Git repository in /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git/\nCloning into 'public'...\nwarning: You appear to have cloned an empty repository.\ndone.\n[master (root-commit) c1c95c0] one\n Author: A U Thor \n 1 file changed, 1 insertion(+)\n create mode 100644 file\nCounting objects: 3, done.\nUnpacking objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 219 bytes, done.\nTotal 3 (delta 0), reused 0 (delta 0)\nTo /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n * [new branch] master -> master\nok 1 - setup repository\n\nexpecting success:\n git clone \"testgit::${PWD}/server\" localclone &&\n test_cmp public/file localclone/file\n\nCloning into 'localclone'...\nok 2 - cloning from local repo\n\nexpecting success:\n git clone \"testgit::file://${PWD}/server\" clone &&\n test_cmp public/file clone/file\n\nCloning into 'clone'...\nok 3 - cloning from remote repo\n\nexpecting success:\n (cd public &&\n echo content >>file &&\n git commit -a -m two &&\n git push)\n\n[master df9eab7] two\n Author: A U Thor \n 1 file changed, 1 insertion(+)\nCounting objects: 5, done.\nUnpacking objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 250 bytes, done.\nTotal 3 (delta 0), reused 0 (delta 0)\nTo /devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n c1c95c0..df9eab7 master -> master\nok 4 - create new commit on remote\n\nexpecting success:\n (cd localclone && git pull) &&\n test_cmp public/file localclone/file\n\n>From testgit::/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n c1c95c0..df9eab7 master -> origin/master\nUpdating c1c95c0..df9eab7\nFast-forward\n file | 1 +\n 1 file changed, 1 insertion(+)\nok 5 - pulling from local repo\n\nexpecting success:\n (cd clone && git pull) &&\n test_cmp public/file clone/file\n\n>From testgit::file:///devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server\n c1c95c0..df9eab7 master -> origin/master\nUpdating c1c95c0..df9eab7\nFast-forward\n file | 1 +\n 1 file changed, 1 insertion(+)\nok 6 - pulling from remote remote\n\nexpecting success:\n (cd localclone &&\n echo content >>file &&\n git commit -a -m three &&\n GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n compare_refs localclone HEAD server HEAD\n\n[master 4dcddca] three\n Author: A U Thor \n 1 file changed, 1 insertion(+)\nDebug: Remote helper: -> capabilities\nDebug: Remote helper: Waiting...\nprefix: 'refs/testgit/origin/'\nGot arguments ['origin', '/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server']\nGot command 'capabilities' with args ''\nDebug: Remote helper: <- import\nDebug: Got cap import\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- export\nDebug: Got cap export\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- refspec refs/heads/*:refs/testgit/origin/*\nDebug: Got cap refspec refs/heads/*:refs/testgit/origin/*\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- *export-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Got cap export-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- *import-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Got cap import-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <-\nDebug: Capabilities complete.\nDebug: Remote helper: Waiting...\nGot command 'list' with args ''\n? refs/heads/master\n@refs/heads/master HEAD\nDebug: Remote helper: <- ? refs/heads/master\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- @refs/heads/master HEAD\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <-\nDebug: Read ref listing.\nDebug: Remote helper: -> export\nGot command 'export' with args ''\nDebug: Remote helper: Waiting...\n\n-*-*-\n\nThe test hangs at this point.\n\nThanks,\n Stefano\n"},{"id":"189310","messageId":"20120415011118.GA4123@ecki","threadId":"30236","inReplyTo":"4F8A0F96.5060408@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-15T01:11:20Z","receivedAt":"2012-04-15T01:11:20Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sun, Apr 15, 2012 at 02:00:22AM +0200, Stefano Lattarini wrote:\n>\n> Debug: Remote helper: -> export\n> Got command 'export' with args ''\n> Debug: Remote helper: Waiting...\n\nHmm. Looks like fast-import and fast-export are in a deadlock. It's\nprobably time to let the experts take over. But if you want to indulge\nme further, you could try again with GIT_TRACE=1. That should give you\nthe arguments to use for the fast-export and fast-import commands right\nbefore it hangs. Then run ./t5800-remote-helpers.sh -i with the\nfollowing change to make it abort right before git push:\n\ndiff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\nindex 1c62001..c4855e9 100755\n--- a/t/t5800-remote-helpers.sh\n+++ b/t/t5800-remote-helpers.sh\n@@ -68,7 +68,8 @@ test_expect_success 'pushing to local repo' '\n (cd localclone &&\n echo content >>file &&\n git commit -a -m three &&\n- git push) &&\n+ false &&\n+ GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n compare_refs localclone HEAD server HEAD\n '\n \n\nYou can manually now run fast-export using the arguments from the first\nrun, for example in my case:\n\n cd trash\\ directory.t5800-remote-helpers\n (cd localclone; git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '--import-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master') |\n\nThat should output:\n\nfeature done\nblob\nmark :5\ndata 24\ncontent\ncontent\ncontent\n\ncommit refs/heads/master\nmark :6\nauthor A U Thor 1334451823 +0000\ncommitter C O Mitter 1334451823 +0000\ndata 6\nthree\nfrom :4\nM 100644 :5 file\n\ndone\n\n\nFinally, you can manually pipe fast-export to fast-import, again using\nthe arguments from the first run:\n\n cd trash\\ directory.t5800-remote-helpers\n (cd localclone; git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '--import-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master') |\n (cd server; git fast-import '--quiet' '--export-marks=/home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks' '--import-marks=/home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks')\n\nIf the fast-export output is the same, and if the last command also\nhangs for you, then we can start debugging fast-import.\n\nClemens\n"},{"id":"189317","messageId":"4F8A8211.2010908@gmail.com","threadId":"30236","inReplyTo":"20120415011118.GA4123@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T08:08:49Z","receivedAt":"2012-04-15T08:08:49Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/15/2012 03:11 AM, Clemens Buchacher wrote:\n> But if you want to indulge\n> me further, you could try again with GIT_TRACE=1. That should give you\n> the arguments to use for the fast-export and fast-import commands right\n> before it hangs.\n\nHere is the output:\n\n-*-*-\n\nexpecting success:\n (cd localclone &&\n echo content >>file &&\n git commit -a -m three &&\n GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n compare_refs localclone HEAD server HEAD\n\n[master 813bd24] three\n Author: A U Thor \n 1 file changed, 1 insertion(+)\ntrace: built-in: git 'push'\ntrace: run_command: 'git-remote-testgit' 'origin' '/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server'\nDebug: Remote helper: -> capabilities\nDebug: Remote helper: Waiting...\ntrace: built-in: git 'ls-remote' '/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git'\ntrace: run_command: 'git-upload-pack '\\''/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git'\\'''\ntrace: exec: 'sh' '-c' 'git-upload-pack '\\''/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git'\\''' 'git-upload-pack '\\''/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server/.git'\\'''\nprefix: 'refs/testgit/origin/'\nGot arguments ['origin', '/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/server']\nGot command 'capabilities' with args ''\nDebug: Remote helper: <- import\nDebug: Got cap import\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- export\nDebug: Got cap export\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- refspec refs/heads/*:refs/testgit/origin/*\nDebug: Got cap refspec refs/heads/*:refs/testgit/origin/*\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- *export-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Got cap export-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- *import-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Got cap import-marks .git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <-\nDebug: Capabilities complete.\nDebug: Remote helper: Waiting...\nGot command 'list' with args ''\n? refs/heads/master\n@refs/heads/master HEAD\nDebug: Remote helper: <- ? refs/heads/master\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <- @refs/heads/master HEAD\nDebug: Remote helper: Waiting...\nDebug: Remote helper: <-\nDebug: Read ref listing.\nDebug: Remote helper: -> export\ntrace: run_command: 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\ntrace: exec: 'git' 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\ntrace: built-in: git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\nGot command 'export' with args ''\ntrace: built-in: git 'for-each-ref' 'refs/heads'\nDebug: Remote helper: Waiting...\ntrace: exec: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\ntrace: run_command: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n\n-*-*-\n\n> Then run ./t5800-remote-helpers.sh -i with the\n> following change to make it abort right before git push\n\nWith this, the I get:\n\n ok 1 - setup repository\n ok 2 - cloning from local repo\n ok 3 - cloning from remote repo\n ok 4 - create new commit on remote\n ok 5 - pulling from local repo\n ok 6 - pulling from remote remote\n not ok - 7 pushing to local repo\n #\n # (cd localclone &&\n # echo content >>file &&\n # git commit -a -m three &&\n # false &&\n # GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n # compare_refs localclone HEAD server HEAD\n #\n\nThanks,\n Stefano\n"},{"id":"189327","messageId":"20120415105943.GD6263@ecki","threadId":"30236","inReplyTo":"4F8A8211.2010908@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-15T10:59:45Z","receivedAt":"2012-04-15T10:59:45Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sun, Apr 15, 2012 at 10:08:49AM +0200, Stefano Lattarini wrote:\n>\n> trace: run_command: 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\n>\n> trace: run_command: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n\nOk, so the above two commands seem to be deadlocking for you. Could you\nplease run git fast-export with the arguments above from the directory\nt/trash directory.t5800-remote-helpers/server and check the output? In\nmy previous email I also gave further instructions on how to pipe\nfast-export output to fast-import. Maybe you missed it, or is anything\nunclear?\n"},{"id":"189329","messageId":"4F8AAD0C.4060207@gmail.com","threadId":"30236","inReplyTo":"20120415011118.GA4123@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T11:12:12Z","receivedAt":"2012-04-15T11:12:12Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/15/2012 03:11 AM, Clemens Buchacher wrote:\n> On Sun, Apr 15, 2012 at 02:00:22AM +0200, Stefano Lattarini wrote:\n>>\n>> Debug: Remote helper: -> export\n>> Got command 'export' with args ''\n>> Debug: Remote helper: Waiting...\n> \n> Hmm. Looks like fast-import and fast-export are in a deadlock. It's\n> probably time to let the experts take over. But if you want to indulge\n> me further, you could try again with GIT_TRACE=1. That should give you\n> the arguments to use for the fast-export and fast-import commands right\n> before it hangs. Then run ./t5800-remote-helpers.sh -i with the\n> following change to make it abort right before git push:\n> \n> diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\n> index 1c62001..c4855e9 100755\n> --- a/t/t5800-remote-helpers.sh\n> +++ b/t/t5800-remote-helpers.sh\n> @@ -68,7 +68,8 @@ test_expect_success 'pushing to local repo' '\n> (cd localclone &&\n> echo content >>file &&\n> git commit -a -m three &&\n> - git push) &&\n> + false &&\n> + GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n> compare_refs localclone HEAD server HEAD\n> '\n> \n> \nAs your later reply pointed out: yes, I had missed the remaining part of your\nmessage :-( Sorry for the confusion. I'm going to reply to it now.\n\n> You can manually now run fast-export using the arguments from the first\n> run, for example in my case:\n> \n> cd trash\\ directory.t5800-remote-helpers\n> (cd localclone; git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '--import-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master') |\n> \n> That should output:\n> \n> feature done\n> blob\n> mark :5\n> data 24\n> content\n> content\n> content\n> \n> commit refs/heads/master\n> mark :6\n> author A U Thor 1334451823 +0000\n> committer C O Mitter 1334451823 +0000\n> data 6\n> three\n> from :4\n> M 100644 :5 file\n> \n> done\n> \n>\nNope, for me it fails with status '128' and with this error message:\n\n fatal: cannot read '.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks': No such file or directory\n\n> Finally, you can manually pipe fast-export to fast-import, again using\n> the arguments from the first run:\n> \n> cd trash\\ directory.t5800-remote-helpers\n> (cd localclone; git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '--import-marks=.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master') |\n> (cd server; git fast-import '--quiet' '--export-marks=/home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks' '--import-marks=/home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks')\n> \nThis fails with the following message:\n\nfatal: cannot read '.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/testgit.marks': No such file or directory\nfatal: cannot read '/home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks': No such file or directory\nfast-import: dumping crash report to .git/fast_import_crash_19819\nerror: Unable to write marks file /home/drizzd/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/cbf054fee258a261ee43ebdc835f0da71b3d1562/git.marks: No such file or directory\n\n> If the fast-export output is the same, and if the last command also\n> hangs for you, then we can start debugging fast-import.\n> \n> Clemens\n\nThanks,\n Stefano\n"},{"id":"189330","messageId":"4F8AAE7C.1020507@gmail.com","threadId":"30236","inReplyTo":"20120415105943.GD6263@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T11:18:20Z","receivedAt":"2012-04-15T11:18:20Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/15/2012 12:59 PM, Clemens Buchacher wrote:\n> On Sun, Apr 15, 2012 at 10:08:49AM +0200, Stefano Lattarini wrote:\n>>\n>> trace: run_command: 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\n>>\n>> trace: run_command: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n> \n> Ok, so the above two commands seem to be deadlocking for you. Could you\n> please run git fast-export with the arguments above from the directory\n> t/trash directory.t5800-remote-helpers/server and check the output?\n>\n\n>From the 'localclone' subdirectory:\n\n$ git 'fast-export' '--use-done-feature' \\\n '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n '^refs/testgit/origin/master' 'refs/heads/master')\n\n feature done\n blob\n mark :5\n data 24\n content\n content\n content\n\n commit refs/heads/master\n mark :6\n author A U Thor 1334477289 +0000\n committer C O Mitter 1334477289 +0000\n data 6\n three\n from :4\n M 100644 :5 file\n\n done\n\n$ git 'fast-export' '--use-done-feature' \\\n '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n '^refs/testgit/origin/master' 'refs/heads/master' \\\n | git 'fast-import' \\\n '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n\n git-fast-import statistics:\n ---------------------------------------------------------------------\n Alloc'd objects: 5000\n Total objects: 0 ( 0 duplicates )\n blobs : 0 ( 0 duplicates 0 deltas of 0 attempts)\n trees : 0 ( 0 duplicates 0 deltas of 0 attempts)\n commits: 0 ( 0 duplicates 0 deltas of 0 attempts)\n tags : 0 ( 0 duplicates 0 deltas of 0 attempts)\n Total branches: 0 ( 0 loads )\n marks: 1024 ( 4 unique )\n atoms: 0\n Memory total: 2243 KiB\n pools: 2048 KiB\n objects: 195 KiB\n ---------------------------------------------------------------------\n pack_report: getpagesize() = 4096\n pack_report: core.packedGitWindowSize = 33554432\n pack_report: core.packedGitLimit = 268435456\n pack_report: pack_used_ctr = 4\n pack_report: pack_mmap_calls = 2\n pack_report: pack_open_windows = 2 / 2\n pack_report: pack_mapped = 471 / 471\n ---------------------------------------------------------------------\n\nBoth the commands are successful.\n\nThanks,\n Stefano\n"},{"id":"189331","messageId":"20120415114518.GB9338@ecki","threadId":"30236","inReplyTo":"4F8AAE7C.1020507@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-15T11:45:18Z","receivedAt":"2012-04-15T11:45:18Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"Hi Stefano,\n\nAlmost there. :-)\n\nOn Sun, Apr 15, 2012 at 01:18:20PM +0200, Stefano Lattarini wrote:\n> \n> $ git 'fast-export' '--use-done-feature' \\\n> '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> '^refs/testgit/origin/master' 'refs/heads/master' \\\n> | git 'fast-import' \\\n> '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n> '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n\nTo simulate a git push, you have to run fast-export from localclone and\nfast-import from the server directory. I.e. starting out in\nt/trash directory.t5800-remote-helpers:\n\n$ (cd localclone; git fast-export --use-done-feature \\\n --export-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n --import-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n ^refs/testgit/origin/master refs/heads/master) |\n (cd server; git fast-import \\\n --export-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n --import-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks')\n"},{"id":"189333","messageId":"4F8AB7F1.1020705@gmail.com","threadId":"30236","inReplyTo":"20120415114518.GB9338@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T11:58:41Z","receivedAt":"2012-04-15T11:58:41Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/15/2012 01:45 PM, Clemens Buchacher wrote:\n> Hi Stefano,\n> \n> Almost there. :-)\n> \n> On Sun, Apr 15, 2012 at 01:18:20PM +0200, Stefano Lattarini wrote:\n>>\n>> $ git 'fast-export' '--use-done-feature' \\\n>> '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n>> '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n>> '^refs/testgit/origin/master' 'refs/heads/master' \\\n>> | git 'fast-import' \\\n>> '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n>> '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n> \n> To simulate a git push, you have to run fast-export from localclone and\n> fast-import from the server directory. I.e. starting out in\n> t/trash directory.t5800-remote-helpers:\n> \n> $ (cd localclone; git fast-export --use-done-feature \\\n> --export-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> --import-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> ^refs/testgit/origin/master refs/heads/master) |\n> (cd server; git fast-import \\\n> --export-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n> --import-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks')\n>\nOK, I cut & pasted your command; it ran successfully, with this output:\n\ngit-fast-import statistics:\n---------------------------------------------------------------------\nAlloc'd objects: 5000\nTotal objects: 3 ( 0 duplicates )\n blobs : 1 ( 0 duplicates 0 deltas of 1 attempts)\n trees : 1 ( 0 duplicates 0 deltas of 1 attempts)\n commits: 1 ( 0 duplicates 0 deltas of 0 attempts)\n tags : 0 ( 0 duplicates 0 deltas of 0 attempts)\nTotal branches: 1 ( 1 loads )\n marks: 1024 ( 6 unique )\n atoms: 1\nMemory total: 2294 KiB\n pools: 2098 KiB\n objects: 195 KiB\n---------------------------------------------------------------------\npack_report: getpagesize() = 4096\npack_report: core.packedGitWindowSize = 33554432\npack_report: core.packedGitLimit = 268435456\npack_report: pack_used_ctr = 2\npack_report: pack_mmap_calls = 1\npack_report: pack_open_windows = 1 / 1\npack_report: pack_mapped = 253 / 253\n---------------------------------------------------------------------\n\nThanks,\n Stefano\n"},{"id":"189334","messageId":"20120415120901.GA14003@ecki","threadId":"30236","inReplyTo":"4F8AB7F1.1020705@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-15T12:09:01Z","receivedAt":"2012-04-15T12:09:01Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sun, Apr 15, 2012 at 01:58:41PM +0200, Stefano Lattarini wrote:\n>\n> OK, I cut & pasted your command; it ran successfully, with this output:\n\nAnd your installed git version is the same as the one you are seeing the\ndeadlock with?\n"},{"id":"189336","messageId":"20120415125140.GA15933@ecki","threadId":"30236","inReplyTo":"4F8AB7F1.1020705@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-15T12:51:41Z","receivedAt":"2012-04-15T12:51:41Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"Hi Sverre,\n\nThe 'pushing to local repo' test of t5800 which pushes using\ngit-remote-testgit.py results in a deadlock for Stefano. Invoking\nfast-export/fast-import manually seems to work, however. I don't know\nwhat else to try at this point.\n\nI am pasting some relevant results below.\n\n> expecting success:\n> (cd localclone &&\n> echo content >>file &&\n> git commit -a -m three &&\n> GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n> compare_refs localclone HEAD server HEAD\n> \n> [...]\n> Debug: Remote helper: -> export\n> trace: run_command: 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed\n> 7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\n> trace: exec: 'git' 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\n> trace: built-in: git 'fast-export' '--use-done-feature' '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' '^refs/testgit/origin/master' 'refs/heads/master'\n> Got command 'export' with args ''\n> trace: built-in: git 'for-each-ref' 'refs/heads'\n> Debug: Remote helper: Waiting...\n> trace: exec: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n> trace: run_command: 'git-fast-import' '--quiet' '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n\nThis is Stefano's output, at which point git fast-import hangs. On my\nbox, fast-export runs a little later (after Got command 'export' with\nargs ''), but I don't see how that would matter.\n\nBelow is the result from running fast-import/fast-export manually:\n\nOn Sun, Apr 15, 2012 at 01:58:41PM +0200, Stefano Lattarini wrote:\n> On 04/15/2012 01:45 PM, Clemens Buchacher wrote:\n> > Hi Stefano,\n> > \n> > Almost there. :-)\n> > \n> > On Sun, Apr 15, 2012 at 01:18:20PM +0200, Stefano Lattarini wrote:\n> >>\n> >> $ git 'fast-export' '--use-done-feature' \\\n> >> '--export-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> >> '--import-marks=.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> >> '^refs/testgit/origin/master' 'refs/heads/master' \\\n> >> | git 'fast-import' \\\n> >> '--export-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n> >> '--import-marks=/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks'\n> > \n> > To simulate a git push, you have to run fast-export from localclone and\n> > fast-import from the server directory. I.e. starting out in\n> > t/trash directory.t5800-remote-helpers:\n> > \n> > $ (cd localclone; git fast-export --use-done-feature \\\n> > --export-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> > --import-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n> > ^refs/testgit/origin/master refs/heads/master) |\n> > (cd server; git fast-import \\\n> > --export-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n> > --import-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks')\n> >\n> OK, I cut & pasted your command; it ran successfully, with this output:\n> \n> git-fast-import statistics:\n> ---------------------------------------------------------------------\n> Alloc'd objects: 5000\n> Total objects: 3 ( 0 duplicates )\n> blobs : 1 ( 0 duplicates 0 deltas of 1 attempts)\n> trees : 1 ( 0 duplicates 0 deltas of 1 attempts)\n> commits: 1 ( 0 duplicates 0 deltas of 0 attempts)\n> tags : 0 ( 0 duplicates 0 deltas of 0 attempts)\n> Total branches: 1 ( 1 loads )\n> marks: 1024 ( 6 unique )\n> atoms: 1\n> Memory total: 2294 KiB\n> pools: 2098 KiB\n> objects: 195 KiB\n> ---------------------------------------------------------------------\n> pack_report: getpagesize() = 4096\n> pack_report: core.packedGitWindowSize = 33554432\n> pack_report: core.packedGitLimit = 268435456\n> pack_report: pack_used_ctr = 2\n> pack_report: pack_mmap_calls = 1\n> pack_report: pack_open_windows = 1 / 1\n> pack_report: pack_mapped = 253 / 253\n> ---------------------------------------------------------------------\n> \n> Thanks,\n> Stefano\n> \n> Other possibly relevant information:\n> \n> $ hg --version | head -1\n> Mercurial Distributed SCM (version 1.8.3) $ python --version\n> \n> $ python --version\n> Python 2.7.2+\n> \n> $ /bin/sh --version | head -1\n> GNU bash, version 4.1.5(1)-release (i486-pc-linux-gnu)\n> \n> $ git describe\n> v1.7.10-167-gf245c01\n\nClemens\n"},{"id":"189337","messageId":"4F8ACAC5.7050600@gmail.com","threadId":"30236","inReplyTo":"20120415120901.GA14003@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2012-04-15T13:19:01Z","receivedAt":"2012-04-15T13:19:01Z","isPatch":false,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 04/15/2012 02:09 PM, Clemens Buchacher wrote:\n> On Sun, Apr 15, 2012 at 01:58:41PM +0200, Stefano Lattarini wrote:\n>>\n>> OK, I cut & pasted your command; it ran successfully, with this output:\n> \n> And your installed git version is the same as the one you are seeing the\n> deadlock with?\n>\nNo, but I see the same behaviour if I put '~/src/git/bin-wrappers' early\nin PATH:\n\n$ git --version\ngit version 1.7.10.130.g36e6c\n\n$ (cd localclone; git fast-export --use-done-feature \\\n --export-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n --import-marks='.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/testgit.marks' \\\n ^refs/testgit/origin/master refs/heads/master) \\\n | (cd server; git fast-import \\\n --export-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks' \\\n --import-marks='/devel/stefano/src/git/t/trash directory.t5800-remote-helpers/localclone/.git/info/fast-import/c6c0871f9a7f12cd3f51aa290fbed7c49f539a8c/git.marks')\n\ngit-fast-import statistics:\n---------------------------------------------------------------------\nAlloc'd objects: 5000\nTotal objects: 3 ( 0 duplicates )\n blobs : 1 ( 0 duplicates 0 deltas of 1 attempts)\n trees : 1 ( 0 duplicates 0 deltas of 1 attempts)\n commits: 1 ( 0 duplicates 0 deltas of 0 attempts)\n tags : 0 ( 0 duplicates 0 deltas of 0 attempts)\nTotal branches: 1 ( 1 loads )\n marks: 1024 ( 6 unique )\n atoms: 1\nMemory total: 2294 KiB\n pools: 2098 KiB\n objects: 195 KiB\n---------------------------------------------------------------------\npack_report: getpagesize() = 4096\npack_report: core.packedGitWindowSize = 33554432\npack_report: core.packedGitLimit = 268435456\npack_report: pack_used_ctr = 1\npack_report: pack_mmap_calls = 1\npack_report: pack_open_windows = 1 / 1\npack_report: pack_mapped = 254 / 254\n---------------------------------------------------------------------\n\nRegards,\n Stefano\n"},{"id":"189499","messageId":"CAGdFq_ga1+aPi6buBtRgm3pt+_nic9xn0arBPQbmDn8=Q=S5KA@mail.gmail.com","threadId":"30236","inReplyTo":"20120415125140.GA15933@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2012-04-17T01:46:07Z","receivedAt":"2012-04-17T01:46:07Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Sun, Apr 15, 2012 at 07:51, Clemens Buchacher wrote:\n> The 'pushing to local repo' test of t5800 which pushes using\n> git-remote-testgit.py results in a deadlock for Stefano. Invoking\n> fast-export/fast-import manually seems to work, however. I don't know\n> what else to try at this point.\n>\n> I am pasting some relevant results below.\n\nJunio ran into the same problem before, so far no-one has been able to\nfigure out why this is happening and/or under what circumstances.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"189735","messageId":"20120419233445.GA20790@padd.com","threadId":"30236","inReplyTo":"20120415125140.GA15933@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-19T23:34:45Z","receivedAt":"2012-04-19T23:34:45Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"drizzd@aon.at wrote on Sun, 15 Apr 2012 14:51 +0200:\n> The 'pushing to local repo' test of t5800 which pushes using\n> git-remote-testgit.py results in a deadlock for Stefano. Invoking\n> fast-export/fast-import manually seems to work, however. I don't know\n> what else to try at this point.\n> \n> I am pasting some relevant results below.\n> \n> > expecting success:\n> > (cd localclone &&\n> > echo content >>file &&\n> > git commit -a -m three &&\n> > GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n> > compare_refs localclone HEAD server HEAD\n\nfast-export should say to fd 1 \"feature done\\nblob\\n...\"\n\nfast-import is probably blocked on reading fd 0, hoping to find a\nsequence of commands\n\nI too cannot reproduce the problem. Stefano, can you try this\nbigger hammer? We'll see if anything sticks out.\n\n GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 strace -vf -s 2000 -o /tmp/strace.out git push\n\nThen \"bzip2 /tmp/strace.out\" and attach /tmp/strace.out.bz2, please.\n\n\t\t-- Pete\n"},{"id":"189814","messageId":"20120421201524.GA18419@padd.com","threadId":"30236","inReplyTo":"4F9145A1.6020201@gmail.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-21T20:15:24Z","receivedAt":"2012-04-21T20:15:24Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"stefano.lattarini@gmail.com wrote on Fri, 20 Apr 2012 13:16 +0200:\n> On 04/20/2012 01:34 AM, Pete Wyckoff wrote:\n> > drizzd@aon.at wrote on Sun, 15 Apr 2012 14:51 +0200:\n> >> The 'pushing to local repo' test of t5800 which pushes using\n> >> git-remote-testgit.py results in a deadlock for Stefano. Invoking\n> >> fast-export/fast-import manually seems to work, however. I don't know\n> >> what else to try at this point.\n> >>\n> >> I am pasting some relevant results below.\n> >>\n> >>> expecting success:\n> >>> (cd localclone &&\n> >>> echo content >>file &&\n> >>> git commit -a -m three &&\n> >>> GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 git push) &&\n> >>> compare_refs localclone HEAD server HEAD\n> > \n> > fast-export should say to fd 1 \"feature done\\nblob\\n...\"\n> > \n> > fast-import is probably blocked on reading fd 0, hoping to find a\n> > sequence of commands\n> > \n> > I too cannot reproduce the problem. Stefano, can you try this\n> > bigger hammer? We'll see if anything sticks out.\n> > \n> > GIT_TRACE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_TESTGIT=1 strace -vf -s 2000 -o /tmp/strace.out git push\n> > \n> > Then \"bzip2 /tmp/strace.out\" and attach /tmp/strace.out.bz2, please.\n> > \n> > \t\t-- Pete\n> >\n> Here it is. Details of the git version I've used to do this test:\n> \n> $ git describe\n> v1.7.10-207-g0bb2ee1\n> \n> $ git diff\n> diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\n> index 1c62001..770cda6 100755\n> --- a/t/t5800-remote-helpers.sh\n> +++ b/t/t5800-remote-helpers.sh\n> @@ -68,7 +68,10 @@ test_expect_success 'pushing to local repo' '\n> (cd localclone &&\n> echo content >>file &&\n> git commit -a -m three &&\n> - git push) &&\n> + GIT_TRACE=1 \\\n> + GIT_TRANSPORT_HELPER_DEBUG=1 \\\n> + GIT_DEBUG_TESTGIT=1 \\\n> + strace -vf -s 2000 -o /tmp/strace.out git push) &&\n> compare_refs localclone HEAD server HEAD\n> '\n\nThank you. Adding the strace made that test work, and a later\none hung with the same race condition. Here's a patch that\ndescribes the race, and adds a strategic sleep to make it easily\nreproducible.\n\nNot sure yet what is the right way to fix this. And I don't know\nif it's just git-remote-testgit that has the problem because it\nforks an importer, or if other remote helpers would be similarly\naffected.\n\nThe easiest fix is to invoke \"python -u\" in\ngit-remote-testgit.py, or do\n\n sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n\nBut I'm not sure if this works on all systems. We could also\nconsider adding a handshake after \"export\", to give the helper\ntime to get ready.\n\n\t\t-- Pete\n\n--8<--\n\n>From 4179e326d86b6ab9f9c7f79a9dbd03b8d81b6a6a Mon Sep 17 00:00:00 2001\nFrom: Pete Wyckoff \nDate: Sat, 21 Apr 2012 14:57:31 -0400\nSubject: [PATCH] demonstrate git-remote-testgit race\n\nTest t5800-remote-helpers \"pushing to local repo\" can hang\nin a race condition between git push and git-remote-testgit.\n\nOn the writer side, \"git push\" invokes push_refs_with_export(),\nwhich sends to stdout the command \"export\\n\" and immediately\nstarts up \"git fast-export\". The latter writes its output stream\nto the same stdout.\n\nOn the reader side, remote helper \"git-remote-testgit\" reads from\nstdin to get its next command. It uses getc() to read characters\nfrom libc up until \\n. Libc has buffered a potentially much\nlarger chunk of stdin. When it sees the \"export\\n\" command, it\nforks \"git fast-import\" to read the stream.\n\nIf fast-export is faster than git-remote-testgit + git\nfast-import, the fast-export output can end up in libc's buffer,\nrather than in git fast-import. The latter blocks on a now-empty\nstdin.\n\nThis patch demonstates the problem.\n\nSigned-off-by: Pete Wyckoff \n---\n git-remote-testgit.py | 7 +++++++\n t/t5800-remote-helpers.sh | 17 +++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex 3dc4851..616e772 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -22,6 +22,7 @@ except ImportError:\n _digest = sha.new\n import sys\n import os\n+import time\n sys.path.insert(0, os.getenv(\"GITPYTHONLIB\",\".\"))\n \n from git_remote_helpers.util import die, debug, warn\n@@ -204,6 +205,12 @@ def read_one_line(repo):\n \"\"\"Reads and processes one command.\n \"\"\"\n \n+ sleepy = os.environ.get(\"GIT_REMOTE_TESTGIT_SLEEPY\")\n+ if sleepy:\n+ debug(\"Sleeping %d sec before readline\" % int(sleepy))\n+ time.sleep(int(sleepy))\n+ debug(\"Awake, to readline\")\n+\n line = sys.stdin.readline()\n \n cmdline = line\ndiff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\nindex 1c62001..7555be6 100755\n--- a/t/t5800-remote-helpers.sh\n+++ b/t/t5800-remote-helpers.sh\n@@ -72,6 +72,23 @@ test_expect_success 'pushing to local repo' '\n \tcompare_refs localclone HEAD server HEAD\n '\n \n+test_expect_failure 'racily pushing to local repo' '\n+\tcp -a server server2 &&\n+\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n+\ttest_when_finished \"rm -rf server2 localclone2\" &&\n+\t(cd localclone2 &&\n+\techo content >>file &&\n+\tgit commit -a -m three &&\n+\tGIT_TRANSPORT_HELPER_DEBUG=1 GIT_REMOTE_TESTGIT_SLEEPY=2 timeout 10 strace -o trace.out -f git push\n+\tret=$? &&\n+\tif [ $ret -eq 124 ]; then\n+\t echo \"Check for too much input in export line\" &&\n+\t grep \"read(0, .export\" trace.out ; true\n+\tfi &&\n+\texit $ret) &&\n+\tcompare_refs localclone2 HEAD server2 HEAD\n+'\n+\n test_expect_success 'synch with changes from localclone' '\n \t(cd clone &&\n \t git pull)\n-- \n1.7.10.57.g7fa2a.dirty\n"},{"id":"189820","messageId":"20120421233521.GB20473@ecki","threadId":"30236","inReplyTo":"20120421201524.GA18419@padd.com","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-21T23:35:21Z","receivedAt":"2012-04-21T23:35:21Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"Hi Pete,\n\nThanks for solving the mystery.\n\nOn Sat, Apr 21, 2012 at 04:15:24PM -0400, Pete Wyckoff wrote:\n> \n> Not sure yet what is the right way to fix this. And I don't know\n> if it's just git-remote-testgit that has the problem because it\n> forks an importer, or if other remote helpers would be similarly\n> affected.\n> \n> The easiest fix is to invoke \"python -u\" in\n> git-remote-testgit.py, or do\n> \n> sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n> \n> But I'm not sure if this works on all systems.\n\nWhich part are you worried about? Maybe the patch below is more\nportable?\n\n> We could also consider adding a handshake after \"export\", to give the\n> helper time to get ready.\n\nNot sure I understand. The helper is throwing away some input. It seems\nto me there is nothing one can do to prevent such brokeness.\n\nIt would be nice if the deadlock could be detected and if git would time\nout in such a case, instead of hanging forever. But rather long delays\ncan also be the result of the remote helper legimitately waiting for\ndata from a slow network.\n\nClemens\n---\n\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex 616e772..bdd1116 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -135,6 +135,22 @@ def update_local_repo(repo):\n return repo\n \n \n+def read_line_fd(fd):\n+ \"\"\"Read from fd until end of line or EOF, and no further.\n+ \"\"\"\n+\n+ line = ''\n+\n+ while True:\n+ c = os.read(fd, 1)\n+ if c == '':\n+ break\n+ line += c\n+ if c == '\\n':\n+ break\n+\n+ return line\n+\n def do_import(repo, args):\n \"\"\"Exports a fast-import stream from testgit for git to import.\n \"\"\"\n@@ -149,7 +165,7 @@ def do_import(repo, args):\n refs = [ref]\n \n while True:\n- line = sys.stdin.readline()\n+ line = read_line_fd(sys.stdin.fileno())\n if line == '\\n':\n break\n if not line.startswith('import '):\n@@ -211,9 +227,7 @@ def read_one_line(repo):\n time.sleep(int(sleepy))\n debug(\"Awake, to readline\")\n \n- line = sys.stdin.readline()\n-\n- cmdline = line\n+ cmdline = read_line_fd(sys.stdin.fileno())\n \n if not cmdline:\n warn(\"Unexpected EOF\")\n-- \n1.7.10\n"},{"id":"189821","messageId":"20120421234211.GC20473@ecki","threadId":"30236","inReplyTo":"20120421234555.GA11808@padd.com","subject":"Re: [PATCH] git-remote-testgit: fix race when spawning fast-import","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-21T23:42:12Z","receivedAt":"2012-04-21T23:42:12Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sat, Apr 21, 2012 at 07:45:55PM -0400, Pete Wyckoff wrote:\n>\n> @@ -258,6 +264,7 @@ def main(args):\n> \n> more = True\n> \n> + sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n> while (more):\n> more = read_one_line(repo)\n\nThere's another read in do_import(). I am not sure if it also suffers\nfrom this race condition, but it certainly has the potential.\n\n> +test_expect_success 'racily pushing to local repo' '\n> +\tcp -a server server2 &&\n> +\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n> +\ttest_when_finished \"rm -rf server2 localclone2\" &&\n> +\t(cd localclone2 &&\n> +\techo content >>file &&\n> +\tgit commit -a -m three &&\n> +\tGIT_REMOTE_TESTGIT_SLEEPY=2 git push) &&\n> +\tcompare_refs localclone2 HEAD server2 HEAD\n> +'\n\nShould this test be running unconditionally? It will delay the otherwise\nalmost instant test by 2 seconds.\n"},{"id":"189819","messageId":"20120421234555.GA11808@padd.com","threadId":"30236","inReplyTo":"20120421201524.GA18419@padd.com","subject":"[PATCH] git-remote-testgit: fix race when spawning fast-import","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-21T23:45:55Z","receivedAt":"2012-04-21T23:45:55Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"Test \"pushing to local repo\" in t5800-remote-helpers can hang\nin a race condition between git push and git-remote-testgit.\n\nOn the writer side, \"git push\" invokes push_refs_with_export(),\nwhich sends to stdout the command \"export\\n\" and immediately\nstarts up \"git fast-export\". The latter writes its output stream\nto the same stdout.\n\nOn the reader side, remote helper \"git-remote-testgit\" reads from\nstdin to get its next command. It uses getc() to read characters\nfrom libc up until \\n. Libc has buffered a potentially much\nlarger chunk of stdin. When it sees the \"export\\n\" command, it\nforks \"git fast-import\" to read the stream.\n\nIf fast-export finishes before git fast-import starts, the\nfast-export output can end up in libc's buffer in\ngit-remote-testgit, rather than in git fast-import. The latter\nhangs indefinitely on a now-empty stdin.\n\nFix this by setting stdin to unbuffered.\n\nSigned-off-by: Pete Wyckoff \n---\n\nThis is the one that might be suitable for upstream. It fixes\nthe problem with this particular remote helper.\n\n git-remote-testgit.py | 7 +++++++\n t/t5800-remote-helpers.sh | 11 +++++++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex 3dc4851..5f3ebd2 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -22,6 +22,7 @@ except ImportError:\n _digest = sha.new\n import sys\n import os\n+import time\n sys.path.insert(0, os.getenv(\"GITPYTHONLIB\",\".\"))\n \n from git_remote_helpers.util import die, debug, warn\n@@ -204,6 +205,11 @@ def read_one_line(repo):\n \"\"\"Reads and processes one command.\n \"\"\"\n \n+ sleepy = os.environ.get(\"GIT_REMOTE_TESTGIT_SLEEPY\")\n+ if sleepy:\n+ debug(\"Sleeping %d sec before readline\" % int(sleepy))\n+ time.sleep(int(sleepy))\n+\n line = sys.stdin.readline()\n \n cmdline = line\n@@ -258,6 +264,7 @@ def main(args):\n \n more = True\n \n+ sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n while (more):\n more = read_one_line(repo)\n \ndiff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\nindex 1c62001..cdcee83 100755\n--- a/t/t5800-remote-helpers.sh\n+++ b/t/t5800-remote-helpers.sh\n@@ -72,6 +72,17 @@ test_expect_success 'pushing to local repo' '\n \tcompare_refs localclone HEAD server HEAD\n '\n \n+test_expect_success 'racily pushing to local repo' '\n+\tcp -a server server2 &&\n+\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n+\ttest_when_finished \"rm -rf server2 localclone2\" &&\n+\t(cd localclone2 &&\n+\techo content >>file &&\n+\tgit commit -a -m three &&\n+\tGIT_REMOTE_TESTGIT_SLEEPY=2 git push) &&\n+\tcompare_refs localclone2 HEAD server2 HEAD\n+'\n+\n test_expect_success 'synch with changes from localclone' '\n \t(cd clone &&\n \t git pull)\n-- \n1.7.10.57.g22c0b.dirty\n"},{"id":"189822","messageId":"20120422021608.GA11892@padd.com","threadId":"30236","inReplyTo":"20120421234211.GC20473@ecki","subject":"Re: [PATCH] git-remote-testgit: fix race when spawning fast-import","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-22T02:16:08Z","receivedAt":"2012-04-22T02:16:08Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"drizzd@aon.at wrote on Sun, 22 Apr 2012 01:42 +0200:\n> On Sat, Apr 21, 2012 at 07:45:55PM -0400, Pete Wyckoff wrote:\n> >\n> > @@ -258,6 +264,7 @@ def main(args):\n> > \n> > more = True\n> > \n> > + sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n> > while (more):\n> > more = read_one_line(repo)\n> \n> There's another read in do_import(). I am not sure if it also suffers\n> from this race condition, but it certainly has the potential.\n\nIt just takes one reopen of stdin for the whole program. So that\none is handled automatically too.\n\n> > +test_expect_success 'racily pushing to local repo' '\n> > +\tcp -a server server2 &&\n> > +\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n> > +\ttest_when_finished \"rm -rf server2 localclone2\" &&\n> > +\t(cd localclone2 &&\n> > +\techo content >>file &&\n> > +\tgit commit -a -m three &&\n> > +\tGIT_REMOTE_TESTGIT_SLEEPY=2 git push) &&\n> > +\tcompare_refs localclone2 HEAD server2 HEAD\n> > +'\n> \n> Should this test be running unconditionally? It will delay the otherwise\n> almost instant test by 2 seconds.\n\nMore like 6 seconds, in fact, for all three interactions in \"git\npush\". I'll change the test to add a prereq on TOOSLOW; is there\na better way to leave a test in to document the issue, but have\nit skipped?\n\n\t\t-- Pete\n"},{"id":"189823","messageId":"20120422021753.GB11892@padd.com","threadId":"30236","inReplyTo":"20120421233521.GB20473@ecki","subject":"Re: master: t5800-remote-helpers.sh hangs on test \"pulling from remote remote\"","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-22T02:17:53Z","receivedAt":"2012-04-22T02:17:53Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"drizzd@aon.at wrote on Sun, 22 Apr 2012 01:35 +0200:\n> Hi Pete,\n> \n> Thanks for solving the mystery.\n> \n> On Sat, Apr 21, 2012 at 04:15:24PM -0400, Pete Wyckoff wrote:\n> > \n> > Not sure yet what is the right way to fix this. And I don't know\n> > if it's just git-remote-testgit that has the problem because it\n> > forks an importer, or if other remote helpers would be similarly\n> > affected.\n> > \n> > The easiest fix is to invoke \"python -u\" in\n> > git-remote-testgit.py, or do\n> > \n> > sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n> > \n> > But I'm not sure if this works on all systems.\n> \n> Which part are you worried about? Maybe the patch below is more\n> portable?\n\nYour way works. But it doesn't handle universal newlines. I\ndon't know if the windows platforms turn that on for stdio. In\nfact, I'm not sure we should be worried about sys.stdin buffering\non windows at all.\n\n> > We could also consider adding a handshake after \"export\", to give the\n> > helper time to get ready.\n> \n> Not sure I understand. The helper is throwing away some input. It seems\n> to me there is nothing one can do to prevent such brokeness.\n\nThe handshake would give the helper time to get its streams\nstraightened up. It would do something like:\n\n read \"export\\n\"\n fork importer, listening to stdin\n write \"export-ready\\n\" to stdout\n\nAnd git push would wait for the \"export-ready\" line before it\nunleases the stream from fast-export. But as Sverre said, that's\na protocol change, which is more difficult. And maybe not\nwarranted if we think this can also be attributed to helper\nbrokenness.\n\nI think we should reopen stdin like this to fix the problem now.\nI'll investigate if it exists on windows too, and see if it is\npossible to fix there too.\n\n> It would be nice if the deadlock could be detected and if git would time\n> out in such a case, instead of hanging forever. But rather long delays\n> can also be the result of the remote helper legimitately waiting for\n> data from a slow network.\n\nTough, generally.\n\n\t\t-- Pete\n\n> Clemens\n> ---\n> \n> diff --git a/git-remote-testgit.py b/git-remote-testgit.py\n> index 616e772..bdd1116 100644\n> --- a/git-remote-testgit.py\n> +++ b/git-remote-testgit.py\n> @@ -135,6 +135,22 @@ def update_local_repo(repo):\n> return repo\n> \n> \n> +def read_line_fd(fd):\n> + \"\"\"Read from fd until end of line or EOF, and no further.\n> + \"\"\"\n> +\n> + line = ''\n> +\n> + while True:\n> + c = os.read(fd, 1)\n> + if c == '':\n> + break\n> + line += c\n> + if c == '\\n':\n> + break\n> +\n> + return line\n> +\n> def do_import(repo, args):\n> \"\"\"Exports a fast-import stream from testgit for git to import.\n> \"\"\"\n> @@ -149,7 +165,7 @@ def do_import(repo, args):\n> refs = [ref]\n> \n> while True:\n> - line = sys.stdin.readline()\n> + line = read_line_fd(sys.stdin.fileno())\n> if line == '\\n':\n> break\n> if not line.startswith('import '):\n> @@ -211,9 +227,7 @@ def read_one_line(repo):\n> time.sleep(int(sleepy))\n> debug(\"Awake, to readline\")\n> \n> - line = sys.stdin.readline()\n> -\n> - cmdline = line\n> + cmdline = read_line_fd(sys.stdin.fileno())\n> \n> if not cmdline:\n> warn(\"Unexpected EOF\")\n> -- \n> 1.7.10\n> \n"},{"id":"189828","messageId":"xmqqty0cxtcd.fsf@junio.mtv.corp.google.com","threadId":"30236","inReplyTo":"20120421234555.GA11808@padd.com","subject":"Re: [PATCH] git-remote-testgit: fix race when spawning fast-import","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-04-22T04:50:58Z","receivedAt":"2012-04-22T04:50:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Pete Wyckoff writes:\n\n> Fix this by setting stdin to unbuffered.\n>\n> Signed-off-by: Pete Wyckoff \n> ---\n>\n> This is the one that might be suitable for upstream. It fixes\n> the problem with this particular remote helper.\n>\n> git-remote-testgit.py | 7 +++++++\n> t/t5800-remote-helpers.sh | 11 +++++++++++\n> 2 files changed, 18 insertions(+)\n>\n> diff --git a/git-remote-testgit.py b/git-remote-testgit.py\n> index 3dc4851..5f3ebd2 100644\n> --- a/git-remote-testgit.py\n> +++ b/git-remote-testgit.py\n> @@ -22,6 +22,7 @@ except ImportError:\n> _digest = sha.new\n> import sys\n> import os\n> +import time\n> sys.path.insert(0, os.getenv(\"GITPYTHONLIB\",\".\"))\n> \n> from git_remote_helpers.util import die, debug, warn\n> @@ -204,6 +205,11 @@ def read_one_line(repo):\n> \"\"\"Reads and processes one command.\n> \"\"\"\n> \n> + sleepy = os.environ.get(\"GIT_REMOTE_TESTGIT_SLEEPY\")\n> + if sleepy:\n> + debug(\"Sleeping %d sec before readline\" % int(sleepy))\n> + time.sleep(int(sleepy))\n> +\n\nIf I understand your explanation correctly, the primary purpose of the\nremote-testgit is to test the parts of the system that talk to remote\nhelpers that are used in production in the t/t5800 script, and this\n\"sleep\" is to make it easier to trigger the particular bug you fixed \nin *this* script. The bug is _not_ in the parts of the system being\ntested, but is in this test scaffolding.\n\nIf that is the case, then it should not be enabled unconditionally.\nWhen somebody wants to see if remote-testgit was broken again (perhaps\nafter observing occassional hangs), the environment should be set when\nrunning the test, but not in t5800.\n"},{"id":"189831","messageId":"20120422094934.GA12132@ecki","threadId":"30236","inReplyTo":"20120422021608.GA11892@padd.com","subject":"Re: [PATCH] git-remote-testgit: fix race when spawning fast-import","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2012-04-22T09:49:34Z","receivedAt":"2012-04-22T09:49:34Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sat, Apr 21, 2012 at 10:16:08PM -0400, Pete Wyckoff wrote:\n> drizzd@aon.at wrote on Sun, 22 Apr 2012 01:42 +0200:\n> > \n> > Should this test be running unconditionally? It will delay the otherwise\n> > almost instant test by 2 seconds.\n> \n> More like 6 seconds, in fact, for all three interactions in \"git\n> push\". I'll change the test to add a prereq on TOOSLOW; is there\n> a better way to leave a test in to document the issue, but have\n> it skipped?\n\nSounds good to me.\n"},{"id":"189844","messageId":"20120422203058.GA17290@padd.com","threadId":"30236","inReplyTo":"xmqqty0cxtcd.fsf@junio.mtv.corp.google.com","subject":"[PATCHv2] git-remote-testgit: fix race when spawning fast-import","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-22T20:30:58Z","receivedAt":"2012-04-22T20:30:58Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"Test \"pushing to local repo\" in t5800-remote-helpers can hang\ndue to a race condition in git-remote-testgit. Fix it by\nsetting stdin to unbuffered.\n\nOn the writer side, \"git push\" invokes push_refs_with_export(),\nwhich sends to stdout the command \"export\\n\" and immediately\nstarts up \"git fast-export\". The latter writes its output stream\nto the same stdout.\n\nOn the reader side, remote helper \"git-remote-testgit\" reads from\nstdin to get its next command. It uses getc() to read characters\nfrom libc up until \\n. Libc has buffered a potentially much\nlarger chunk of stdin. When it sees the \"export\\n\" command, it\nforks \"git fast-import\" to read the stream.\n\nIf fast-export finishes before git fast-import starts, the\nfast-export output can end up in libc's buffer in\ngit-remote-testgit, rather than in git fast-import. The latter\nhangs indefinitely on a now-empty stdin.\n\nSigned-off-by: Pete Wyckoff \n---\ngitster@pobox.com wrote on Sat, 21 Apr 2012 21:50 -0700:\n> If I understand your explanation correctly, the primary purpose of the\n> remote-testgit is to test the parts of the system that talk to remote\n> helpers that are used in production in the t/t5800 script, and this\n> \"sleep\" is to make it easier to trigger the particular bug you fixed \n> in *this* script. The bug is _not_ in the parts of the system being\n> tested, but is in this test scaffolding.\n\nIndeed. I tried to make that more obvious in the commit message.\n\n> If that is the case, then it should not be enabled unconditionally.\n> When somebody wants to see if remote-testgit was broken again (perhaps\n> after observing occassional hangs), the environment should be set when\n> running the test, but not in t5800.\n\nClemens suggested disabling the test by default, as I've done\nhere. I think it would be okay to remove it entirely, too.\n\n\t\t-- Pete\n\n git-remote-testgit.py | 7 +++++++\n t/t5800-remote-helpers.sh | 14 ++++++++++++++\n 2 files changed, 21 insertions(+)\n\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex 3dc4851..5f3ebd2 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -22,6 +22,7 @@ except ImportError:\n _digest = sha.new\n import sys\n import os\n+import time\n sys.path.insert(0, os.getenv(\"GITPYTHONLIB\",\".\"))\n \n from git_remote_helpers.util import die, debug, warn\n@@ -204,6 +205,11 @@ def read_one_line(repo):\n \"\"\"Reads and processes one command.\n \"\"\"\n \n+ sleepy = os.environ.get(\"GIT_REMOTE_TESTGIT_SLEEPY\")\n+ if sleepy:\n+ debug(\"Sleeping %d sec before readline\" % int(sleepy))\n+ time.sleep(int(sleepy))\n+\n line = sys.stdin.readline()\n \n cmdline = line\n@@ -258,6 +264,7 @@ def main(args):\n \n more = True\n \n+ sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n while (more):\n more = read_one_line(repo)\n \ndiff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh\nindex 1c62001..85a8042 100755\n--- a/t/t5800-remote-helpers.sh\n+++ b/t/t5800-remote-helpers.sh\n@@ -72,6 +72,20 @@ test_expect_success 'pushing to local repo' '\n \tcompare_refs localclone HEAD server HEAD\n '\n \n+# Generally, skip this test. It demonstrates a now-fixed\n+# race in git-remote-testgit, but is too slow to leave in\n+# for general use.\n+test_expect_success DEBUG_TESTGIT_RACE 'racily pushing to local repo' '\n+\tcp -a server server2 &&\n+\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n+\ttest_when_finished \"rm -rf server2 localclone2\" &&\n+\t(cd localclone2 &&\n+\techo content >>file &&\n+\tgit commit -a -m three &&\n+\tGIT_REMOTE_TESTGIT_SLEEPY=2 git push) &&\n+\tcompare_refs localclone2 HEAD server2 HEAD\n+'\n+\n test_expect_success 'synch with changes from localclone' '\n \t(cd clone &&\n \t git pull)\n-- \n1.7.10.57.g437cb.dirty\n"},{"id":"189846","messageId":"xmqq4nsbmapw.fsf@junio.mtv.corp.google.com","threadId":"30236","inReplyTo":"20120422203058.GA17290@padd.com","subject":"Re: [PATCHv2] git-remote-testgit: fix race when spawning fast-import","fromName":"Junio C Hamano","fromEmail":"junio@pobox.com","sentAt":"2012-04-23T02:40:59Z","receivedAt":"2012-04-23T02:40:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Pete Wyckoff writes:\n\n> Clemens suggested disabling the test by default, as I've done\n> here. I think it would be okay to remove it entirely, too.\n\nOk.\n\n> +# Generally, skip this test. It demonstrates a now-fixed\n> +# race in git-remote-testgit, but is too slow to leave in\n> +# for general use.\n> +test_expect_success DEBUG_TESTGIT_RACE 'racily pushing to local repo' '\n> +\tcp -a server server2 &&\n> +\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n> +\ttest_when_finished \"rm -rf server2 localclone2\" &&\n\nShouldn't test_when_finished come before these two commands? Otherwise\nserver2 or localclone2 directories will be left behind when they fail\nbefore control reaches it...\n"},{"id":"189862","messageId":"20120423113505.GA22513@padd.com","threadId":"30236","inReplyTo":"xmqq4nsbmapw.fsf@junio.mtv.corp.google.com","subject":"Re: [PATCHv2] git-remote-testgit: fix race when spawning fast-import","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-04-23T11:35:05Z","receivedAt":"2012-04-23T11:35:05Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"junio@pobox.com wrote on Sun, 22 Apr 2012 19:40 -0700:\n> Pete Wyckoff writes:\n> \n> > Clemens suggested disabling the test by default, as I've done\n> > here. I think it would be okay to remove it entirely, too.\n> \n> Ok.\n> \n> > +# Generally, skip this test. It demonstrates a now-fixed\n> > +# race in git-remote-testgit, but is too slow to leave in\n> > +# for general use.\n> > +test_expect_success DEBUG_TESTGIT_RACE 'racily pushing to local repo' '\n> > +\tcp -a server server2 &&\n> > +\tgit clone \"testgit::${PWD}/server2\" localclone2 &&\n> > +\ttest_when_finished \"rm -rf server2 localclone2\" &&\n> \n> Shouldn't test_when_finished come before these two commands? Otherwise\n> server2 or localclone2 directories will be left behind when they fail\n> before control reaches it...\n\nYes, you're right.\n\n\t\t-- Pete\n"}]}