{"thread":{"id":"37159","subject":"Why is git submodule slow under windows","startedAt":"2014-07-18T10:14:12Z","lastAt":"2014-07-18T18:01:52Z","messageCount":4,"participants":["Armbruster Joachim (BEG/EMS1)","Thomas Braun","Fredrik Gustafsson","Jens Lehmann"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"246300","messageId":"18956e25b6c14091a64af5a65ce5d13c@FE-MBX1009.de.bosch.com","threadId":"37159","inReplyTo":null,"subject":"Why is git submodule slow under windows","fromName":"Armbruster Joachim (BEG/EMS1)","fromEmail":"joachim.armbruster@de.bosch.com","sentAt":"2014-07-18T10:14:12Z","receivedAt":"2014-07-18T10:14:12Z","isPatch":false,"sender":{"key":"joachim.armbruster@de.bosch.com","avatar":null},"body":"Hello,\n\nWe split a monolithic repository into ~50 submodules. The stored data has the same size. In the 1:1 comparison to the monolithic repository, the submodule handling is very slow. Under Linux everything remains fast, but windows is slow.\n\nSo, why is git getting slow when it has to deal with a lot of submodules?\nI read something about the lack of the underlying cygwin to handle NTFS in a efficient way. Is this the root cause, or are there other causes also?\n"},{"id":"246304","messageId":"53C90E44.6030702@virtuell-zuhause.de","threadId":"37159","inReplyTo":"18956e25b6c14091a64af5a65ce5d13c@FE-MBX1009.de.bosch.com","subject":"Re: Why is git submodule slow under windows","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2014-07-18T12:08:36Z","receivedAt":"2014-07-18T12:08:36Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):\n> Hello,\n> \n> We split a monolithic repository into ~50 submodules. The stored data\n> has the same size. In the 1:1 comparison to the monolithic\n> repository, the submodule handling is very slow. Under Linux\n> everything remains fast, but windows is slow.\n> \n> So, why is git getting slow when it has to deal with a lot of\n> submodules? I read something about the lack of the underlying cygwin\n> to handle NTFS in a efficient way. Is this the root cause, or are\n> there other causes also?\n> \n\nHi,\n\nI assume you are using the latetst git from https://msysgit.github.io on\nwindows.\n\nI would guess that submodules on windows are slow because\ngit-submodules.sh is a shell script, and bash on windows is not really\nthat fast.\n\nThere has been some (albeit older) discussion on the msysgit\nmailinglist, see [1].\n\nYou can play around with core.fscache [2] maybe that helps.\n\nThomas\n\n[1]: https://groups.google.com/d/msg/msysgit/AuPA4EbwchU/N42tsb6GousJ\n[2]:\nhttps://github.com/msysgit/msysgit/releases/tag/Git-1.8.5.2-preview20131230\n"},{"id":"246307","messageId":"20140718130241.GD14720@paksenarrion.iveqy.com","threadId":"37159","inReplyTo":"53C90E44.6030702@virtuell-zuhause.de","subject":"Re: Why is git submodule slow under windows","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2014-07-18T13:02:41Z","receivedAt":"2014-07-18T13:02:41Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:\n> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):\n> > Hello,\n> > \n> > We split a monolithic repository into ~50 submodules. The stored data\n> > has the same size. In the 1:1 comparison to the monolithic\n> > repository, the submodule handling is very slow. Under Linux\n> > everything remains fast, but windows is slow.\n> > \n> > So, why is git getting slow when it has to deal with a lot of\n> > submodules? I read something about the lack of the underlying cygwin\n> > to handle NTFS in a efficient way. Is this the root cause, or are\n> > there other causes also?\n> > \n> \n> Hi,\n> \n> I assume you are using the latetst git from https://msysgit.github.io on\n> windows.\n> \n> I would guess that submodules on windows are slow because\n> git-submodules.sh is a shell script, and bash on windows is not really\n> that fast.\n\nMy guess is that because the shell script uses fork() heavily and fork()\nis an expensive operation on Windows, that alone causes the slowddown.\n\nI did a quick test a while back when I rewrote part of git-submodule.sh\nin lua and runned it on my repo with ~45 submodules. The speedup was\nsignificant and should be even bigger on windows.\n\n-- \nMed vänlig hälsning\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"246324","messageId":"53C96110.7020801@web.de","threadId":"37159","inReplyTo":"20140718130241.GD14720@paksenarrion.iveqy.com","subject":"Re: Why is git submodule slow under windows","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-07-18T18:01:52Z","receivedAt":"2014-07-18T18:01:52Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 18.07.2014 15:02, schrieb Fredrik Gustafsson:\n> On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:\n>> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):\n>>> Hello,\n>>>\n>>> We split a monolithic repository into ~50 submodules. The stored data\n>>> has the same size. In the 1:1 comparison to the monolithic\n>>> repository, the submodule handling is very slow. Under Linux\n>>> everything remains fast, but windows is slow.\n>>>\n>>> So, why is git getting slow when it has to deal with a lot of\n>>> submodules? I read something about the lack of the underlying cygwin\n>>> to handle NTFS in a efficient way. Is this the root cause, or are\n>>> there other causes also?\n>>>\n>>\n>> Hi,\n>>\n>> I assume you are using the latetst git from https://msysgit.github.io on\n>> windows.\n>>\n>> I would guess that submodules on windows are slow because\n>> git-submodules.sh is a shell script, and bash on windows is not really\n>> that fast.\n> \n> My guess is that because the shell script uses fork() heavily and fork()\n> is an expensive operation on Windows, that alone causes the slowddown.\n> \n> I did a quick test a while back when I rewrote part of git-submodule.sh\n> in lua and runned it on my repo with ~45 submodules. The speedup was\n> significant and should be even bigger on windows.\n\nWithout having looked at your Lua rewrite I suspect my recursive\nsubmodule checkout series could speed that up even more, as it\nonly needs to fork a \"git status\" followed by a \"git checkout\" or\n\"git read-tree\" for each submodule. Now that my submodule test\nharness and Heiko's submodule config lookup API did hit next,\nthis is the thing I'm currently working on.\n"}]}