{"thread":{"id":"36005","subject":"[BUG] Halt during fetch on MacOS","startedAt":"2014-02-28T23:26:28Z","lastAt":"2014-03-06T23:12:27Z","messageCount":11,"participants":["Conley Owens","Jeff King","Kyle J. McKay","Max Horn","Jacopo Notarstefano","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"235677","messageId":"CAFFUb6X455R4OD5FKnVFHFmvTyRqtV300bc=a8Xs03agM+=uLQ@mail.gmail.com","threadId":"36005","inReplyTo":null,"subject":"[BUG] Halt during fetch on MacOS","fromName":"Conley Owens","fromEmail":"cco3@android.com","sentAt":"2014-02-28T23:26:28Z","receivedAt":"2014-02-28T23:26:28Z","isPatch":false,"sender":{"key":"cco3@android.com","avatar":null},"body":"$ git --version  # This is just the git from MacPorts\ngit version 1.8.5.5\n$ sw_vers\nProductName:    Mac OS X\nProductVersion: 10.8.5\nBuildVersion:   12F45\n\ntest.sh\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n#!/bin/bash\nrungit() {\n    mkdir $1\n    GIT_DIR=$1 git init --bare\n    echo '[remote \"aosp\"]' > $1/config\n    echo '    url =\nhttps://android.googlesource.com/platform/external/tinyxml2' >>\n$1/config\n    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n    rm -rf $1\n}\n\nfor i in $(seq 1 100)\ndo\n    rungit testdir$i &\ndone\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n$ ./test.sh  # Warning! This script fetches ~40MB of data\n\nWhen everything cools, you can see that there are some fetches hanging\n(typically).\n$ ps | grep 'git fetch'\n...\n63310 ttys004    0:00.01 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n63314 ttys004    0:00.01 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n63319 ttys004    0:00.01 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n63407 ttys004    0:00.00 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n63414 ttys004    0:00.00 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n63420 ttys004    0:00.00 git fetch aosp\n+refs/heads/master:refs/remotes/aosp/master\n...\n\nYou can look at the parent process of each and see that one half\nspawned the other half, or you can look at the environment variables\nfor each to see that there are two processes operating in the same\ndirectory for each directory where there's an issue.\n$ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\nps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\nGIT_DIR=testdir14\nGIT_DIR=testdir14\nGIT_DIR=testdir32\nGIT_DIR=testdir32\nGIT_DIR=testdir47\nGIT_DIR=testdir47\n\nI've searched through the mailing list, but this doesn't seem to be a\nknown issue.  I've only seen this occur on macs (and with a good deal\nof regularity).  It doesn't occur on my Ubuntu box.\n\n~cco3\n"},{"id":"235695","messageId":"20140301061532.GC20397@sigill.intra.peff.net","threadId":"36005","inReplyTo":"CAFFUb6X455R4OD5FKnVFHFmvTyRqtV300bc=a8Xs03agM+=uLQ@mail.gmail.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-01T06:15:32Z","receivedAt":"2014-03-01T06:15:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:\n\n> test.sh\n> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n> #!/bin/bash\n> rungit() {\n>     mkdir $1\n>     GIT_DIR=$1 git init --bare\n>     echo '[remote \"aosp\"]' > $1/config\n>     echo '    url =\n> https://android.googlesource.com/platform/external/tinyxml2' >>\n> $1/config\n>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n\nI don't think this is affecting your test, but you probably want \">>\" to\nappend to the config for the first line, too. Otherwise you are\noverwriting some of git's default settings.\n\n> When everything cools, you can see that there are some fetches hanging\n> (typically).\n> $ ps | grep 'git fetch'\n> ...\n> 63310 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> [...]\n\nI can't reproduce here on Linux. Can you find out what the processes are\ndoing with something like strace?\n\n> You can look at the parent process of each and see that one half\n> spawned the other half, or you can look at the environment variables\n> for each to see that there are two processes operating in the same\n> directory for each directory where there's an issue.\n> $ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\n> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\n> GIT_DIR=testdir14\n> GIT_DIR=testdir14\n> GIT_DIR=testdir32\n> GIT_DIR=testdir32\n> GIT_DIR=testdir47\n> GIT_DIR=testdir47\n\nA fetch will start many sub-processes. Try:\n\n  GIT_TRACE=1 git fetch \\\n    https://android.googlesource.com/platform/external/tinyxml2\n\nwhich shows git-fetch starting the git-remote-https helper, which in\nturn starts git-fetch-pack to do the actual protocol, which uses\ngit-unpack-objects to manage the incoming pack.\n\n-Peff\n"},{"id":"235767","messageId":"F9B4418D-9414-4857-9199-EDBA20B952C6@gmail.com","threadId":"36005","inReplyTo":"20140301061532.GC20397@sigill.intra.peff.net","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Kyle J. McKay","fromEmail":"mackyle@gmail.com","sentAt":"2014-03-02T02:02:54Z","receivedAt":"2014-03-02T02:02:54Z","isPatch":false,"sender":{"key":"mackyle@gmail.com","avatar":"https://avatars.githubusercontent.com/u/813346?v=4"},"body":"On Feb 28, 2014, at 22:15, Jeff King wrote:\n> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:\n>\n>> test.sh\n>> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n>> #!/bin/bash\n>> rungit() {\n>>    mkdir $1\n>>    GIT_DIR=$1 git init --bare\n>>    echo '[remote \"aosp\"]' > $1/config\n>>    echo '    url =\n>> https://android.googlesource.com/platform/external/tinyxml2' >>\n>> $1/config\n>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/ \n>> master\n>\n> I don't think this is affecting your test, but you probably want  \n> \">>\" to\n> append to the config for the first line, too. Otherwise you are\n> overwriting some of git's default settings.\n\nI replaced it with a call to git config in my version.\n\n>> When everything cools, you can see that there are some fetches  \n>> hanging\n>> (typically).\n>> $ ps | grep 'git fetch'\n>> ...\n>> 63310 ttys004    0:00.01 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> [...]\n>\n> I can't reproduce here on Linux. Can you find out what the processes  \n> are\n> doing with something like strace?\n\nI can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.\n\nWhat I mean by mostly is that the very first time I ran the test  \nscript I got approximately 36 of these errors:\n\nfatal: unable to access 'https://android.googlesource.com/platform/external/tinyxml2/' \n: Unknown SSL protocol error in connection to android.googlesource.com: \n443\n\nThe rest of the fetches completed.  That was with Git 1.8.5.1.\n\nHowever, I was never able to reproduce those errors again.  All the  \nsubsequent runs completed all fetches successfully using that same Git  \nversion so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the  \noriginal and another machine.\n\nI am, however NAT'd, so it's possible the NAT was somehow responsible  \nfor the initial 36 failures.\n\nPerhaps you are seeing a similar issue.\n\nYou might try setting these sysctl variables:\n\n# Timeout new TCP connections after 30 seconds instead of 75\nnet.inet.tcp.keepinit=30000\n# Always keep alive TCP connections\nnet.inet.tcp.always_keepalive=1\n# Start keep alive checks after 30 seconds instead of 2 hours\nnet.inet.tcp.keepidle=30000\n# Wait 5 seconds between probes instead of 75\n# Note that 8 probes in a row must fail to drop the connection\nnet.inet.tcp.keepintvl=5000\n\nthen running your test again and see if the hanging git fetch  \nprocesses die with some kind of failed connection error within about  \n70 seconds or so.  With the default sysctl settings, even with Git  \nenabling keep alives, it would likely take a bit over two hours for a  \ndead connection to be noticed.\n\n--Kyle\n"},{"id":"235799","messageId":"3DC821FE-7330-4163-9E73-D3313B74E001@quendi.de","threadId":"36005","inReplyTo":"CAFFUb6X455R4OD5FKnVFHFmvTyRqtV300bc=a8Xs03agM+=uLQ@mail.gmail.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Max Horn","fromEmail":"max@quendi.de","sentAt":"2014-03-02T14:36:03Z","receivedAt":"2014-03-02T14:36:03Z","isPatch":false,"sender":{"key":"max@quendi.de","avatar":"https://avatars.githubusercontent.com/u/241512?v=4"},"body":"\nOn 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:\n\n> $ git --version  # This is just the git from MacPorts\n> git version 1.8.5.5\n> $ sw_vers\n> ProductName:    Mac OS X\n> ProductVersion: 10.8.5\n> BuildVersion:   12F45\n\nI cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.\n\nNote: I tried this both with you original test.sh, and also with a version were I replaced the \">\" by \">>\", as per Jeff's suggestion. It (as predicted) didn't make any difference).\n\nIf this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?\n\n\nCheers,\nMax\n\n> \n> test.sh\n> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n> #!/bin/bash\n> rungit() {\n>    mkdir $1\n>    GIT_DIR=$1 git init --bare\n>    echo '[remote \"aosp\"]' > $1/config\n>    echo '    url =\n> https://android.googlesource.com/platform/external/tinyxml2' >>\n> $1/config\n>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n>    rm -rf $1\n> }\n> \n> for i in $(seq 1 100)\n> do\n>    rungit testdir$i &\n> done\n> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n> $ ./test.sh  # Warning! This script fetches ~40MB of data\n> \n> When everything cools, you can see that there are some fetches hanging\n> (typically).\n> $ ps | grep 'git fetch'\n> ...\n> 63310 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63314 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63319 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63407 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63414 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63420 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> ...\n> \n> You can look at the parent process of each and see that one half\n> spawned the other half, or you can look at the environment variables\n> for each to see that there are two processes operating in the same\n> directory for each directory where there's an issue.\n> $ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\n> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\n> GIT_DIR=testdir14\n> GIT_DIR=testdir14\n> GIT_DIR=testdir32\n> GIT_DIR=testdir32\n> GIT_DIR=testdir47\n> GIT_DIR=testdir47\n> \n> I've searched through the mailing list, but this doesn't seem to be a\n> known issue.  I've only seen this occur on macs (and with a good deal\n> of regularity).  It doesn't occur on my Ubuntu box.\n> \n> ~cco3\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n\n"},{"id":"235947","messageId":"CAFFUb6XE-5+VoZRs=-YSCBJMYoEi8C+KG7Lrr1B9NmmeOJOzEA@mail.gmail.com","threadId":"36005","inReplyTo":"3DC821FE-7330-4163-9E73-D3313B74E001@quendi.de","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Conley Owens","fromEmail":"cco3@android.com","sentAt":"2014-03-03T22:50:46Z","receivedAt":"2014-03-03T22:50:46Z","isPatch":false,"sender":{"key":"cco3@android.com","avatar":null},"body":"On Sun, Mar 2, 2014 at 6:36 AM, Max Horn <max@quendi.de> wrote:\n>\n> On 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:\n>\n>> $ git --version  # This is just the git from MacPorts\n>> git version 1.8.5.5\n>> $ sw_vers\n>> ProductName:    Mac OS X\n>> ProductVersion: 10.8.5\n>> BuildVersion:   12F45\n>\n> I cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.\n>\n> Note: I tried this both with you original test.sh, and also with a version were I replaced the \">\" by \">>\", as per Jeff's suggestion. It (as predicted) didn't make any difference).\n>\n> If this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?\n\nMac mini i7 with 1TB fusion drive and 16GB ram.\n\nIt also seemed to trigger more when grabbing larger repositories.\nReplace\"external/tinyxml2\" with \"docs/source.android.com\" and you\nmight be more likely to trigger the issue.  You can also try\nincreasing the number of processes to spawn from 100 to 150.\n\n>\n>\n> Cheers,\n> Max\n>\n>>\n>> test.sh\n>> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n>> #!/bin/bash\n>> rungit() {\n>>    mkdir $1\n>>    GIT_DIR=$1 git init --bare\n>>    echo '[remote \"aosp\"]' > $1/config\n>>    echo '    url =\n>> https://android.googlesource.com/platform/external/tinyxml2' >>\n>> $1/config\n>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n>>    rm -rf $1\n>> }\n>>\n>> for i in $(seq 1 100)\n>> do\n>>    rungit testdir$i &\n>> done\n>> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n>> $ ./test.sh  # Warning! This script fetches ~40MB of data\n>>\n>> When everything cools, you can see that there are some fetches hanging\n>> (typically).\n>> $ ps | grep 'git fetch'\n>> ...\n>> 63310 ttys004    0:00.01 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> 63314 ttys004    0:00.01 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> 63319 ttys004    0:00.01 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> 63407 ttys004    0:00.00 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> 63414 ttys004    0:00.00 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> 63420 ttys004    0:00.00 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> ...\n>>\n>> You can look at the parent process of each and see that one half\n>> spawned the other half, or you can look at the environment variables\n>> for each to see that there are two processes operating in the same\n>> directory for each directory where there's an issue.\n>> $ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\n>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\n>> GIT_DIR=testdir14\n>> GIT_DIR=testdir14\n>> GIT_DIR=testdir32\n>> GIT_DIR=testdir32\n>> GIT_DIR=testdir47\n>> GIT_DIR=testdir47\n>>\n>> I've searched through the mailing list, but this doesn't seem to be a\n>> known issue.  I've only seen this occur on macs (and with a good deal\n>> of regularity).  It doesn't occur on my Ubuntu box.\n>>\n>> ~cco3\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>>\n>\n"},{"id":"235970","messageId":"CAFFUb6W7htX9zVB669Ki84nef5XE9Uek_aJhHLRwFg25Kt=cDQ@mail.gmail.com","threadId":"36005","inReplyTo":"20140301061532.GC20397@sigill.intra.peff.net","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Conley Owens","fromEmail":"cco3@android.com","sentAt":"2014-03-04T01:03:05Z","receivedAt":"2014-03-04T01:03:05Z","isPatch":false,"sender":{"key":"cco3@android.com","avatar":null},"body":"On Fri, Feb 28, 2014 at 10:15 PM, Jeff King <peff@peff.net> wrote:\n> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:\n>\n>> test.sh\n>> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n>> #!/bin/bash\n>> rungit() {\n>>     mkdir $1\n>>     GIT_DIR=$1 git init --bare\n>>     echo '[remote \"aosp\"]' > $1/config\n>>     echo '    url =\n>> https://android.googlesource.com/platform/external/tinyxml2' >>\n>> $1/config\n>>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n>\n> I don't think this is affecting your test, but you probably want \">>\" to\n> append to the config for the first line, too. Otherwise you are\n> overwriting some of git's default settings.\n>\n>> When everything cools, you can see that there are some fetches hanging\n>> (typically).\n>> $ ps | grep 'git fetch'\n>> ...\n>> 63310 ttys004    0:00.01 git fetch aosp\n>> +refs/heads/master:refs/remotes/aosp/master\n>> [...]\n>\n> I can't reproduce here on Linux. Can you find out what the processes are\n> doing with something like strace?\n\nYes, none of my Linux users have had any problems with this, but many\nof my Mac users have.\n\nI'm trying to run it under dtruss, but it's slowing the entire system\ndown to a halt.\n\n>\n>> You can look at the parent process of each and see that one half\n>> spawned the other half, or you can look at the environment variables\n>> for each to see that there are two processes operating in the same\n>> directory for each directory where there's an issue.\n>> $ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\n>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\n>> GIT_DIR=testdir14\n>> GIT_DIR=testdir14\n>> GIT_DIR=testdir32\n>> GIT_DIR=testdir32\n>> GIT_DIR=testdir47\n>> GIT_DIR=testdir47\n>\n> A fetch will start many sub-processes. Try:\n>\n>   GIT_TRACE=1 git fetch \\\n>     https://android.googlesource.com/platform/external/tinyxml2\n>\n> which shows git-fetch starting the git-remote-https helper, which in\n> turn starts git-fetch-pack to do the actual protocol, which uses\n> git-unpack-objects to manage the incoming pack.\n\nThis is the full output of a fetch that ends up hanging:\ntrace: built-in: git 'fetch' 'aosp'\n'+refs/heads/master:refs/remotes/aosp/master'\ntrace: run_command: 'git-remote-https' 'aosp'\n'https://android.googlesource.com/platform/external/tinyxml2'\n\n>\n> -Peff\n"},{"id":"236047","messageId":"CAL0uuq0MPSnB+HNxfD2Mj6mLbhTcvhKQ_e20EV0QgS8Os8CsJw@mail.gmail.com","threadId":"36005","inReplyTo":"F9B4418D-9414-4857-9199-EDBA20B952C6@gmail.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Jacopo Notarstefano","fromEmail":"jacopo.notarstefano@gmail.com","sentAt":"2014-03-04T22:21:40Z","receivedAt":"2014-03-04T22:21:40Z","isPatch":false,"sender":{"key":"jacopo.notarstefano@gmail.com","avatar":"https://avatars.githubusercontent.com/u/381280?v=4"},"body":"On Sun, Mar 2, 2014 at 3:02 AM, Kyle J. McKay <mackyle@gmail.com> wrote:\n> I can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.\n>\n> What I mean by mostly is that the very first time I ran the test script I\n> got approximately 36 of these errors:\n>\n> fatal: unable to access\n> 'https://android.googlesource.com/platform/external/tinyxml2/': Unknown SSL\n> protocol error in connection to android.googlesource.com:443\n>\n> The rest of the fetches completed.  That was with Git 1.8.5.1.\n>\n> However, I was never able to reproduce those errors again.  All the\n> subsequent runs completed all fetches successfully using that same Git\n> version so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the original\n> and another machine.\n\nWith Git 1.9.0.138.g2de3478 (latest build from source) on Mac OS X\n10.9.2 I report similar results. A first run yielded several  \"fatal:\nunable to access\" errors, while a second run yielded just one.\n"},{"id":"236133","messageId":"CAFFUb6ViYitJC1sjL_nyWY1RHuJKURfcZ+_5Zk2kr+suh+txZg@mail.gmail.com","threadId":"36005","inReplyTo":"CAFFUb6X455R4OD5FKnVFHFmvTyRqtV300bc=a8Xs03agM+=uLQ@mail.gmail.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Conley Owens","fromEmail":"cco3@android.com","sentAt":"2014-03-05T21:56:13Z","receivedAt":"2014-03-05T21:56:13Z","isPatch":false,"sender":{"key":"cco3@android.com","avatar":null},"body":"On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:\n> $ git --version  # This is just the git from MacPorts\n> git version 1.8.5.5\n> $ sw_vers\n> ProductName:    Mac OS X\n> ProductVersion: 10.8.5\n> BuildVersion:   12F45\n\nOK, I've tried using my own build from master, and I still get the same results.\n\nI've done a little more investigation and discovered it always hangs at:\n`atexit(notify_parent);` in `run-command.c:start_command`\nwhen running:\ntrace: run_command: 'git-remote-https' 'aosp'\n'https://android.googlesource.com/platform/external/tinyxml2'\n\nCould this have to do with the atexit implementation?  (eg. limit on\nthe number of functions that can be registered, etc)\n\n$ cc -v\nApple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)\nTarget: x86_64-apple-darwin12.5.0\nThread model: posix\n\n>\n> test.sh\n> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n> #!/bin/bash\n> rungit() {\n>     mkdir $1\n>     GIT_DIR=$1 git init --bare\n>     echo '[remote \"aosp\"]' > $1/config\n>     echo '    url =\n> https://android.googlesource.com/platform/external/tinyxml2' >>\n> $1/config\n>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master\n>     rm -rf $1\n> }\n>\n> for i in $(seq 1 100)\n> do\n>     rungit testdir$i &\n> done\n> \"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n> $ ./test.sh  # Warning! This script fetches ~40MB of data\n>\n> When everything cools, you can see that there are some fetches hanging\n> (typically).\n> $ ps | grep 'git fetch'\n> ...\n> 63310 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63314 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63319 ttys004    0:00.01 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63407 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63414 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> 63420 ttys004    0:00.00 git fetch aosp\n> +refs/heads/master:refs/remotes/aosp/master\n> ...\n>\n> You can look at the parent process of each and see that one half\n> spawned the other half, or you can look at the environment variables\n> for each to see that there are two processes operating in the same\n> directory for each directory where there's an issue.\n> $ echo \"$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do\n> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)\" | sort\n> GIT_DIR=testdir14\n> GIT_DIR=testdir14\n> GIT_DIR=testdir32\n> GIT_DIR=testdir32\n> GIT_DIR=testdir47\n> GIT_DIR=testdir47\n>\n> I've searched through the mailing list, but this doesn't seem to be a\n> known issue.  I've only seen this occur on macs (and with a good deal\n> of regularity).  It doesn't occur on my Ubuntu box.\n>\n> ~cco3\n"},{"id":"236176","messageId":"xmqqk3c7yyku.fsf@gitster.dls.corp.google.com","threadId":"36005","inReplyTo":"CAFFUb6ViYitJC1sjL_nyWY1RHuJKURfcZ+_5Zk2kr+suh+txZg@mail.gmail.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-06T18:24:49Z","receivedAt":"2014-03-06T18:24:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Conley Owens <cco3@android.com> writes:\n\n> On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:\n>> $ git --version  # This is just the git from MacPorts\n>> git version 1.8.5.5\n>> $ sw_vers\n>> ProductName:    Mac OS X\n>> ProductVersion: 10.8.5\n>> BuildVersion:   12F45\n>\n> OK, I've tried using my own build from master, and I still get the same results.\n>\n> I've done a little more investigation and discovered it always hangs at:\n> `atexit(notify_parent);` in `run-command.c:start_command`\n> when running:\n> trace: run_command: 'git-remote-https' 'aosp'\n> 'https://android.googlesource.com/platform/external/tinyxml2'\n>\n> Could this have to do with the atexit implementation?  (eg. limit on\n> the number of functions that can be registered, etc)\n\nThanks.\n\nAn interesting theory indeed.  I read that an implementation is\nsupposed to take at least ATEXIT_MAX (32) calls to atexit(3); while\nI do think we register functions with atexit(3) from multiple places\nin our code, I doubt we would be making that many.\n\n> $ cc -v\n> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)\n> Target: x86_64-apple-darwin12.5.0\n> Thread model: posix\n"},{"id":"236200","messageId":"20140306211645.GA11926@sigill.intra.peff.net","threadId":"36005","inReplyTo":"xmqqk3c7yyku.fsf@gitster.dls.corp.google.com","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-06T21:16:45Z","receivedAt":"2014-03-06T21:16:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:\n\n> > OK, I've tried using my own build from master, and I still get the same results.\n> >\n> > I've done a little more investigation and discovered it always hangs at:\n> > `atexit(notify_parent);` in `run-command.c:start_command`\n> > when running:\n> > trace: run_command: 'git-remote-https' 'aosp'\n> > 'https://android.googlesource.com/platform/external/tinyxml2'\n> >\n> > Could this have to do with the atexit implementation?  (eg. limit on\n> > the number of functions that can be registered, etc)\n> \n> Thanks.\n> \n> An interesting theory indeed.  I read that an implementation is\n> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while\n> I do think we register functions with atexit(3) from multiple places\n> in our code, I doubt we would be making that many.\n\nIt seems awfully weird that it would _hang_ in such a case, though. That\nsounds more like hitting a mutex that's internal to atexit(), or\nsomething similar.\n\nConley, can you see if dropping that atexit clears up the problem (you\nshould be OK without it; git will just fail to notice the child's\nexec failure with as much detail).\n\n-Peff\n"},{"id":"236217","messageId":"CAFFUb6X7NJfQQKkJsM+KWn5hiE0C_t3-_DKZh7tONBCm3XVfRQ@mail.gmail.com","threadId":"36005","inReplyTo":"20140306211645.GA11926@sigill.intra.peff.net","subject":"Re: [BUG] Halt during fetch on MacOS","fromName":"Conley Owens","fromEmail":"cco3@android.com","sentAt":"2014-03-06T23:12:27Z","receivedAt":"2014-03-06T23:12:27Z","isPatch":false,"sender":{"key":"cco3@android.com","avatar":null},"body":"On Thu, Mar 6, 2014 at 1:16 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:\n>\n>> > OK, I've tried using my own build from master, and I still get the same results.\n>> >\n>> > I've done a little more investigation and discovered it always hangs at:\n>> > `atexit(notify_parent);` in `run-command.c:start_command`\n>> > when running:\n>> > trace: run_command: 'git-remote-https' 'aosp'\n>> > 'https://android.googlesource.com/platform/external/tinyxml2'\n>> >\n>> > Could this have to do with the atexit implementation?  (eg. limit on\n>> > the number of functions that can be registered, etc)\n>>\n>> Thanks.\n>>\n>> An interesting theory indeed.  I read that an implementation is\n>> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while\n>> I do think we register functions with atexit(3) from multiple places\n>> in our code, I doubt we would be making that many.\n>\n> It seems awfully weird that it would _hang_ in such a case, though. That\n> sounds more like hitting a mutex that's internal to atexit(), or\n> something similar.\n\nYou are correct that it's a mutex internal to atexit.  The crazy thing\nis that there is only one thread sitting and waiting for it.\n====================\n(gdb) thread apply all bt\n\nThread 1 (process 71053):\n#0  0x00007fff8cb67122 in __psynch_mutexwait ()\n#1  0x00007fff833d0dcd in pthread_mutex_lock ()\n#2  0x00007fff8d4a77e0 in LockHelper::LockHelper ()\n#3  0x00007fff8d4a8d0a in dladdr ()\n#4  0x00007fff83412260 in atexit ()\n#5  0x000000010597b35e in start_command (cmd=0x7fe1cd801b30) at\nrun-command.c:374\n#6  0x0000000105998959 in get_helper (transport=<value temporarily\nunavailable, due to optimizations>) at transport-helper.c:142\n#7  0x00000001059970bd in get_refs_list (transport=0x7fe1cd801370,\nfor_push=0) at transport-helper.c:954\n#8  0x000000010599635d in transport_get_remote_refs\n(transport=0x7fe1cd801370) at transport.c:1227\n#9  0x00000001058b7469 in get_ref_map [inlined] () at\n/Users/android-build/cco3/master/git/builtin/fetch.c:278\n#10 0x00000001058b7469 in fetch_one (remote=<value temporarily\nunavailable, due to optimizations>, argc=<value temporarily\nunavailable, due to optimizations>, argv=0x7fff00000000) at\nbuiltin/fetch.c:862\n#11 0x00000001058b6f22 in cmd_fetch (argc=<value temporarily\nunavailable, due to optimizations>, argv=<value temporarily\nunavailable, due to optimizations>, prefix=0x0) at\nbuiltin/fetch.c:1158\n#12 0x0000000105890acc in run_builtin [inlined] () at\n/Users/android-build/cco3/master/git/git.c:314\n#13 0x0000000105890acc in handle_builtin (argc=3, argv=0x7fff5a370648)\nat git.c:487\n#14 0x00000001058906c1 in main (argc=3, av=<value temporarily\nunavailable, due to optimizations>) at git.c:533\n===================\n\n>\n> Conley, can you see if dropping that atexit clears up the problem (you\n> should be OK without it; git will just fail to notice the child's\n> exec failure with as much detail).\n\nYes, I'm unable to reproduce the issue after dropping atexit.\n\n>\n> -Peff\n"}]}