threads / bug / 36005

[BUG] Halt during fetch on MacOS

Subject: [BUG] Halt during fetch on MacOS

## tl;dr

11 messages between Feb 28, 2014 and Mar 6, 2014.

replies: 10people: 6as markdown or json

Conley Owens· Feb 28, 2014, 23:26 UTC · lore
$ git --version  # This is just the git from MacPorts
git version 1.8.5.5
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.8.5
BuildVersion:   12F45
test.sh
"""""""""""""""""""""""""""""""""""""
#!/bin/bash
rungit() {
    mkdir $1
    GIT_DIR=$1 git init --bare
    echo '[remote "aosp"]' > $1/config
    echo '    url =
https://android.googlesource.com/platform/external/tinyxml2' >>
$1/config
    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
    rm -rf $1
}
for i in $(seq 1 100)
do
    rungit testdir$i &
done
"""""""""""""""""""""""""""""""""""""""
$ ./test.sh  # Warning! This script fetches ~40MB of data

When everything cools, you can see that there are some fetches hanging (typically). $ ps | grep 'git fetch' ... 63310 ttys004 0:00.01 git fetch aosp +refs/heads/master:refs/remotes/aosp/master 63314 ttys004 0:00.01 git fetch aosp +refs/heads/master:refs/remotes/aosp/master 63319 ttys004 0:00.01 git fetch aosp +refs/heads/master:refs/remotes/aosp/master 63407 ttys004 0:00.00 git fetch aosp +refs/heads/master:refs/remotes/aosp/master 63414 ttys004 0:00.00 git fetch aosp +refs/heads/master:refs/remotes/aosp/master 63420 ttys004 0:00.00 git fetch aosp +refs/heads/master:refs/remotes/aosp/master ...

You can look at the parent process of each and see that one half spawned the other half, or you can look at the environment variables for each to see that there are two processes operating in the same directory for each directory where there's an issue. $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort GIT_DIR=testdir14 GIT_DIR=testdir14 GIT_DIR=testdir32 GIT_DIR=testdir32 GIT_DIR=testdir47 GIT_DIR=testdir47

I've searched through the mailing list, but this doesn't seem to be a known issue. I've only seen this occur on macs (and with a good deal of regularity). It doesn't occur on my Ubuntu box.

~cco3
Jeff King· Mar 1, 2014, 06:15 UTC · re: Conley Owens · lore

Re: [BUG] Halt during fetch on MacOS

On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:
Show 11 quoted lines
> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>     mkdir $1
>     GIT_DIR=$1 git init --bare
>     echo '[remote "aosp"]' > $1/config
>     echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master

I don't think this is affecting your test, but you probably want ">>" to append to the config for the first line, too. Otherwise you are overwriting some of git's default settings.

Show 7 quoted lines
> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> [...]

I can't reproduce here on Linux. Can you find out what the processes are doing with something like strace?

Show 12 quoted lines
> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47
A fetch will start many sub-processes. Try:
  GIT_TRACE=1 git fetch \
    https://android.googlesource.com/platform/external/tinyxml2

which shows git-fetch starting the git-remote-https helper, which in turn starts git-fetch-pack to do the actual protocol, which uses git-unpack-objects to manage the incoming pack.

-Peff
Kyle J. McKay· Mar 2, 2014, 02:02 UTC · re: Jeff King · lore

Re: [BUG] Halt during fetch on MacOS

On Feb 28, 2014, at 22:15, Jeff King wrote:
Show 19 quoted lines
> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:
>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>    mkdir $1
>>    GIT_DIR=$1 git init --bare
>>    echo '[remote "aosp"]' > $1/config
>>    echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/ 
>> master
>
> I don't think this is affecting your test, but you probably want  
> ">>" to
> append to the config for the first line, too. Otherwise you are
> overwriting some of git's default settings.
I replaced it with a call to git config in my version.
Show 12 quoted lines
>> When everything cools, you can see that there are some fetches  
>> hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> [...]
>
> I can't reproduce here on Linux. Can you find out what the processes  
> are
> doing with something like strace?
I can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.

What I mean by mostly is that the very first time I ran the test script I got approximately 36 of these errors:

fatal: unable to access 'https://android.googlesource.com/platform/external/tinyxml2/' : Unknown SSL protocol error in connection to android.googlesource.com: 443

The rest of the fetches completed.  That was with Git 1.8.5.1.

However, I was never able to reproduce those errors again. All the subsequent runs completed all fetches successfully using that same Git version so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the original and another machine.

I am, however NAT'd, so it's possible the NAT was somehow responsible for the initial 36 failures.

Perhaps you are seeing a similar issue.
You might try setting these sysctl variables:

# Timeout new TCP connections after 30 seconds instead of 75 net.inet.tcp.keepinit=30000 # Always keep alive TCP connections net.inet.tcp.always_keepalive=1 # Start keep alive checks after 30 seconds instead of 2 hours net.inet.tcp.keepidle=30000 # Wait 5 seconds between probes instead of 75 # Note that 8 probes in a row must fail to drop the connection net.inet.tcp.keepintvl=5000

then running your test again and see if the hanging git fetch processes die with some kind of failed connection error within about 70 seconds or so. With the default sysctl settings, even with Git enabling keep alives, it would likely take a bit over two hours for a dead connection to be noticed.

--Kyle
Jacopo Notarstefano· Mar 4, 2014, 22:21 UTC · re: Kyle J. McKay · lore

Re: [BUG] Halt during fetch on MacOS

On Sun, Mar 2, 2014 at 3:02 AM, Kyle J. McKay <mackyle@gmail.com> wrote:
Show 15 quoted lines
> I can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.
>
> What I mean by mostly is that the very first time I ran the test script I
> got approximately 36 of these errors:
>
> fatal: unable to access
> 'https://android.googlesource.com/platform/external/tinyxml2/': Unknown SSL
> protocol error in connection to android.googlesource.com:443
>
> The rest of the fetches completed.  That was with Git 1.8.5.1.
>
> However, I was never able to reproduce those errors again.  All the
> subsequent runs completed all fetches successfully using that same Git
> version so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the original
> and another machine.

With Git 1.9.0.138.g2de3478 (latest build from source) on Mac OS X 10.9.2 I report similar results. A first run yielded several "fatal: unable to access" errors, while a second run yielded just one.

Conley Owens· Mar 4, 2014, 01:03 UTC · re: Jeff King · lore

Re: [BUG] Halt during fetch on MacOS

On Fri, Feb 28, 2014 at 10:15 PM, Jeff King <peff@peff.net> wrote:
Show 28 quoted lines
> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:
>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>     mkdir $1
>>     GIT_DIR=$1 git init --bare
>>     echo '[remote "aosp"]' > $1/config
>>     echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>
> I don't think this is affecting your test, but you probably want ">>" to
> append to the config for the first line, too. Otherwise you are
> overwriting some of git's default settings.
>
>> When everything cools, you can see that there are some fetches hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> [...]
>
> I can't reproduce here on Linux. Can you find out what the processes are
> doing with something like strace?

Yes, none of my Linux users have had any problems with this, but many of my Mac users have.

I'm trying to run it under dtruss, but it's slowing the entire system down to a halt.

Show 22 quoted lines
>
>> You can look at the parent process of each and see that one half
>> spawned the other half, or you can look at the environment variables
>> for each to see that there are two processes operating in the same
>> directory for each directory where there's an issue.
>> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
>> GIT_DIR=testdir14
>> GIT_DIR=testdir14
>> GIT_DIR=testdir32
>> GIT_DIR=testdir32
>> GIT_DIR=testdir47
>> GIT_DIR=testdir47
>
> A fetch will start many sub-processes. Try:
>
>   GIT_TRACE=1 git fetch \
>     https://android.googlesource.com/platform/external/tinyxml2
>
> which shows git-fetch starting the git-remote-https helper, which in
> turn starts git-fetch-pack to do the actual protocol, which uses
> git-unpack-objects to manage the incoming pack.

This is the full output of a fetch that ends up hanging: trace: built-in: git 'fetch' 'aosp' '+refs/heads/master:refs/remotes/aosp/master' trace: run_command: 'git-remote-https' 'aosp' 'https://android.googlesource.com/platform/external/tinyxml2'

>
> -Peff
Max Horn· Mar 2, 2014, 14:36 UTC · re: Conley Owens · lore

Re: [BUG] Halt during fetch on MacOS

On 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:
Show 6 quoted lines
> $ git --version  # This is just the git from MacPorts
> git version 1.8.5.5
> $ sw_vers
> ProductName:    Mac OS X
> ProductVersion: 10.8.5
> BuildVersion:   12F45
I cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.
Note: I tried this both with you original test.sh, and also with a version were I replaced the ">" by ">>", as per Jeff's suggestion. It (as predicted) didn't make any difference).
If this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?

Cheers, Max

Show 63 quoted lines
> 
> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>    mkdir $1
>    GIT_DIR=$1 git init --bare
>    echo '[remote "aosp"]' > $1/config
>    echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>    rm -rf $1
> }
> 
> for i in $(seq 1 100)
> do
>    rungit testdir$i &
> done
> """""""""""""""""""""""""""""""""""""""
> $ ./test.sh  # Warning! This script fetches ~40MB of data
> 
> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63314 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63319 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63407 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63414 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63420 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> ...
> 
> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47
> 
> I've searched through the mailing list, but this doesn't seem to be a
> known issue.  I've only seen this occur on macs (and with a good deal
> of regularity).  It doesn't occur on my Ubuntu box.
> 
> ~cco3
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Conley Owens· Mar 3, 2014, 22:50 UTC · re: Max Horn · lore

Re: [BUG] Halt during fetch on MacOS

On Sun, Mar 2, 2014 at 6:36 AM, Max Horn <max@quendi.de> wrote:
Show 15 quoted lines
>
> On 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:
>
>> $ git --version  # This is just the git from MacPorts
>> git version 1.8.5.5
>> $ sw_vers
>> ProductName:    Mac OS X
>> ProductVersion: 10.8.5
>> BuildVersion:   12F45
>
> I cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.
>
> Note: I tried this both with you original test.sh, and also with a version were I replaced the ">" by ">>", as per Jeff's suggestion. It (as predicted) didn't make any difference).
>
> If this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?
Mac mini i7 with 1TB fusion drive and 16GB ram.

It also seemed to trigger more when grabbing larger repositories. Replace"external/tinyxml2" with "docs/source.android.com" and you might be more likely to trigger the issue. You can also try increasing the number of processes to spawn from 100 to 150.

Show 69 quoted lines
>
>
> Cheers,
> Max
>
>>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>    mkdir $1
>>    GIT_DIR=$1 git init --bare
>>    echo '[remote "aosp"]' > $1/config
>>    echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>>    rm -rf $1
>> }
>>
>> for i in $(seq 1 100)
>> do
>>    rungit testdir$i &
>> done
>> """""""""""""""""""""""""""""""""""""""
>> $ ./test.sh  # Warning! This script fetches ~40MB of data
>>
>> When everything cools, you can see that there are some fetches hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63314 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63319 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63407 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63414 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63420 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> ...
>>
>> You can look at the parent process of each and see that one half
>> spawned the other half, or you can look at the environment variables
>> for each to see that there are two processes operating in the same
>> directory for each directory where there's an issue.
>> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
>> GIT_DIR=testdir14
>> GIT_DIR=testdir14
>> GIT_DIR=testdir32
>> GIT_DIR=testdir32
>> GIT_DIR=testdir47
>> GIT_DIR=testdir47
>>
>> I've searched through the mailing list, but this doesn't seem to be a
>> known issue.  I've only seen this occur on macs (and with a good deal
>> of regularity).  It doesn't occur on my Ubuntu box.
>>
>> ~cco3
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
Conley Owens· Mar 5, 2014, 21:56 UTC · re: Conley Owens · lore

Re: [BUG] Halt during fetch on MacOS

On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:
Show 6 quoted lines
> $ git --version  # This is just the git from MacPorts
> git version 1.8.5.5
> $ sw_vers
> ProductName:    Mac OS X
> ProductVersion: 10.8.5
> BuildVersion:   12F45
OK, I've tried using my own build from master, and I still get the same results.

I've done a little more investigation and discovered it always hangs at: `atexit(notify_parent);` in `run-command.c:start_command` when running: trace: run_command: 'git-remote-https' 'aosp' 'https://android.googlesource.com/platform/external/tinyxml2'

Could this have to do with the atexit implementation? (eg. limit on the number of functions that can be registered, etc)

$ cc -v
Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix
Show 58 quoted lines
>
> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>     mkdir $1
>     GIT_DIR=$1 git init --bare
>     echo '[remote "aosp"]' > $1/config
>     echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>     rm -rf $1
> }
>
> for i in $(seq 1 100)
> do
>     rungit testdir$i &
> done
> """""""""""""""""""""""""""""""""""""""
> $ ./test.sh  # Warning! This script fetches ~40MB of data
>
> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63314 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63319 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63407 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63414 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63420 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> ...
>
> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47
>
> I've searched through the mailing list, but this doesn't seem to be a
> known issue.  I've only seen this occur on macs (and with a good deal
> of regularity).  It doesn't occur on my Ubuntu box.
>
> ~cco3
Junio C Hamano· Mar 6, 2014, 18:24 UTC · re: Conley Owens · lore

Re: [BUG] Halt during fetch on MacOS

Conley Owens <cco3@android.com> writes:
Show 18 quoted lines
> On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:
>> $ git --version  # This is just the git from MacPorts
>> git version 1.8.5.5
>> $ sw_vers
>> ProductName:    Mac OS X
>> ProductVersion: 10.8.5
>> BuildVersion:   12F45
>
> OK, I've tried using my own build from master, and I still get the same results.
>
> I've done a little more investigation and discovered it always hangs at:
> `atexit(notify_parent);` in `run-command.c:start_command`
> when running:
> trace: run_command: 'git-remote-https' 'aosp'
> 'https://android.googlesource.com/platform/external/tinyxml2'
>
> Could this have to do with the atexit implementation?  (eg. limit on
> the number of functions that can be registered, etc)
Thanks.

An interesting theory indeed. I read that an implementation is supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while I do think we register functions with atexit(3) from multiple places in our code, I doubt we would be making that many.

> $ cc -v
> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
> Target: x86_64-apple-darwin12.5.0
> Thread model: posix
Jeff King· Mar 6, 2014, 21:16 UTC · re: Junio C Hamano · lore

Re: [BUG] Halt during fetch on MacOS

On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:
Show 17 quoted lines
> > OK, I've tried using my own build from master, and I still get the same results.
> >
> > I've done a little more investigation and discovered it always hangs at:
> > `atexit(notify_parent);` in `run-command.c:start_command`
> > when running:
> > trace: run_command: 'git-remote-https' 'aosp'
> > 'https://android.googlesource.com/platform/external/tinyxml2'
> >
> > Could this have to do with the atexit implementation?  (eg. limit on
> > the number of functions that can be registered, etc)
> 
> Thanks.
> 
> An interesting theory indeed.  I read that an implementation is
> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while
> I do think we register functions with atexit(3) from multiple places
> in our code, I doubt we would be making that many.

It seems awfully weird that it would _hang_ in such a case, though. That sounds more like hitting a mutex that's internal to atexit(), or something similar.

Conley, can you see if dropping that atexit clears up the problem (you should be OK without it; git will just fail to notice the child's exec failure with as much detail).

-Peff
Conley Owens· Mar 6, 2014, 23:12 UTC · re: Jeff King · lore

Re: [BUG] Halt during fetch on MacOS

On Thu, Mar 6, 2014 at 1:16 PM, Jeff King <peff@peff.net> wrote:
Show 23 quoted lines
> On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:
>
>> > OK, I've tried using my own build from master, and I still get the same results.
>> >
>> > I've done a little more investigation and discovered it always hangs at:
>> > `atexit(notify_parent);` in `run-command.c:start_command`
>> > when running:
>> > trace: run_command: 'git-remote-https' 'aosp'
>> > 'https://android.googlesource.com/platform/external/tinyxml2'
>> >
>> > Could this have to do with the atexit implementation?  (eg. limit on
>> > the number of functions that can be registered, etc)
>>
>> Thanks.
>>
>> An interesting theory indeed.  I read that an implementation is
>> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while
>> I do think we register functions with atexit(3) from multiple places
>> in our code, I doubt we would be making that many.
>
> It seems awfully weird that it would _hang_ in such a case, though. That
> sounds more like hitting a mutex that's internal to atexit(), or
> something similar.

You are correct that it's a mutex internal to atexit. The crazy thing is that there is only one thread sitting and waiting for it. ==================== (gdb) thread apply all bt

Thread 1 (process 71053): #0 0x00007fff8cb67122 in __psynch_mutexwait () #1 0x00007fff833d0dcd in pthread_mutex_lock () #2 0x00007fff8d4a77e0 in LockHelper::LockHelper () #3 0x00007fff8d4a8d0a in dladdr () #4 0x00007fff83412260 in atexit () #5 0x000000010597b35e in start_command (cmd=0x7fe1cd801b30) at run-command.c:374 #6 0x0000000105998959 in get_helper (transport=<value temporarily unavailable, due to optimizations>) at transport-helper.c:142 #7 0x00000001059970bd in get_refs_list (transport=0x7fe1cd801370, for_push=0) at transport-helper.c:954 #8 0x000000010599635d in transport_get_remote_refs (transport=0x7fe1cd801370) at transport.c:1227 #9 0x00000001058b7469 in get_ref_map [inlined] () at /Users/android-build/cco3/master/git/builtin/fetch.c:278 #10 0x00000001058b7469 in fetch_one (remote=<value temporarily unavailable, due to optimizations>, argc=<value temporarily unavailable, due to optimizations>, argv=0x7fff00000000) at builtin/fetch.c:862 #11 0x00000001058b6f22 in cmd_fetch (argc=<value temporarily unavailable, due to optimizations>, argv=<value temporarily unavailable, due to optimizations>, prefix=0x0) at builtin/fetch.c:1158 #12 0x0000000105890acc in run_builtin [inlined] () at /Users/android-build/cco3/master/git/git.c:314 #13 0x0000000105890acc in handle_builtin (argc=3, argv=0x7fff5a370648) at git.c:487 #14 0x00000001058906c1 in main (argc=3, av=<value temporarily unavailable, due to optimizations>) at git.c:533 ===================

>
> Conley, can you see if dropping that atexit clears up the problem (you
> should be OK without it; git will just fail to notice the child's
> exec failure with as much detail).
Yes, I'm unable to reproduce the issue after dropping atexit.
>
> -Peff

← back to recent threads