threads / discuss / 37159

Why is git submodule slow under windows

Subject: Why is git submodule slow under windows

## tl;dr

4 messages between Jul 18, 2014 and Jul 18, 2014.

replies: 3people: 4as markdown or json

Armbruster Joachim (BEG/EMS1)· Jul 18, 2014, 10:14 UTC · lore

Hello,

We split a monolithic repository into ~50 submodules. The stored data has the same size. In the 1:1 comparison to the monolithic repository, the submodule handling is very slow. Under Linux everything remains fast, but windows is slow.

So, why is git getting slow when it has to deal with a lot of submodules? I read something about the lack of the underlying cygwin to handle NTFS in a efficient way. Is this the root cause, or are there other causes also?

Thomas Braun· Jul 18, 2014, 12:08 UTC · re: Armbruster Joachim (BEG/EMS1) · lore

Re: Why is git submodule slow under windows

Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):

Show 12 quoted lines

> Hello,
> 
> We split a monolithic repository into ~50 submodules. The stored data
> has the same size. In the 1:1 comparison to the monolithic
> repository, the submodule handling is very slow. Under Linux
> everything remains fast, but windows is slow.
> 
> So, why is git getting slow when it has to deal with a lot of
> submodules? I read something about the lack of the underlying cygwin
> to handle NTFS in a efficient way. Is this the root cause, or are
> there other causes also?
>

Hi,

I assume you are using the latetst git from https://msysgit.github.io on windows.

I would guess that submodules on windows are slow because git-submodules.sh is a shell script, and bash on windows is not really that fast.

There has been some (albeit older) discussion on the msysgit mailinglist, see [1].

You can play around with core.fscache [2] maybe that helps.

Thomas

[1]: https://groups.google.com/d/msg/msysgit/AuPA4EbwchU/N42tsb6GousJ [2]: https://github.com/msysgit/msysgit/releases/tag/Git-1.8.5.2-preview20131230

Fredrik Gustafsson· Jul 18, 2014, 13:02 UTC · re: Thomas Braun · lore

Re: Why is git submodule slow under windows

On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:

Show 22 quoted lines

> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):
> > Hello,
> > 
> > We split a monolithic repository into ~50 submodules. The stored data
> > has the same size. In the 1:1 comparison to the monolithic
> > repository, the submodule handling is very slow. Under Linux
> > everything remains fast, but windows is slow.
> > 
> > So, why is git getting slow when it has to deal with a lot of
> > submodules? I read something about the lack of the underlying cygwin
> > to handle NTFS in a efficient way. Is this the root cause, or are
> > there other causes also?
> > 
> 
> Hi,
> 
> I assume you are using the latetst git from https://msysgit.github.io on
> windows.
> 
> I would guess that submodules on windows are slow because
> git-submodules.sh is a shell script, and bash on windows is not really
> that fast.

My guess is that because the shell script uses fork() heavily and fork() is an expensive operation on Windows, that alone causes the slowddown.

I did a quick test a while back when I rewrote part of git-submodule.sh in lua and runned it on my repo with ~45 submodules. The speedup was significant and should be even bigger on windows.

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Jens Lehmann· Jul 18, 2014, 18:01 UTC · re: Fredrik Gustafsson · lore

Re: Why is git submodule slow under windows

Am 18.07.2014 15:02, schrieb Fredrik Gustafsson:

Show 30 quoted lines

> On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:
>> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):
>>> Hello,
>>>
>>> We split a monolithic repository into ~50 submodules. The stored data
>>> has the same size. In the 1:1 comparison to the monolithic
>>> repository, the submodule handling is very slow. Under Linux
>>> everything remains fast, but windows is slow.
>>>
>>> So, why is git getting slow when it has to deal with a lot of
>>> submodules? I read something about the lack of the underlying cygwin
>>> to handle NTFS in a efficient way. Is this the root cause, or are
>>> there other causes also?
>>>
>>
>> Hi,
>>
>> I assume you are using the latetst git from https://msysgit.github.io on
>> windows.
>>
>> I would guess that submodules on windows are slow because
>> git-submodules.sh is a shell script, and bash on windows is not really
>> that fast.
> 
> My guess is that because the shell script uses fork() heavily and fork()
> is an expensive operation on Windows, that alone causes the slowddown.
> 
> I did a quick test a while back when I rewrote part of git-submodule.sh
> in lua and runned it on my repo with ~45 submodules. The speedup was
> significant and should be even bigger on windows.

Without having looked at your Lua rewrite I suspect my recursive submodule checkout series could speed that up even more, as it only needs to fork a "git status" followed by a "git checkout" or "git read-tree" for each submodule. Now that my submodule test harness and Heiko's submodule config lookup API did hit next, this is the thing I'm currently working on.