# THREADED_DELTA_SEARCH 13 messages from 2008-07-10 to 2008-07-11. Participants: Pierre Habouzit, Johannes Sixt, Jakub Narebski, Junio C Hamano, Nicolas Pitre, Andreas Ericsson, Alex Riesen. Thread: https://gitlist.dev/t/14391 ## Pierre Habouzit, 2008-07-10 07:53 Subject: THREADED_DELTA_SEARCH Message-ID: <20080710075327.GD24819@artemis.madism.org> URL: https://gitlist.dev/e/20080710075327.GD24819%40artemis.madism.org ``` I'm using it in production for quite a long time now, and I wonder if there is any specific reason why it's not default. Debian activated it by default too a couple of weeks ago, which means that it's in production on quite a large scale, and there are no issues reported either. Would a patch making it default be accepted ? It's quite handy given that SMP machines are really pervasive nowadays… -- ·O· Pierre Habouzit ··O madcoder@debian.org OOO http://www.madism.org ``` ## Johannes Sixt, 2008-07-10 08:01 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <4875C1CF.1010604@viscovery.net> URL: https://gitlist.dev/e/4875C1CF.1010604%40viscovery.net In-Reply-To: <20080710075327.GD24819@artemis.madism.org> ``` Pierre Habouzit schrieb: > I'm using it in production for quite a long time now, and I wonder if > there is any specific reason why it's not default. Debian activated it > by default too a couple of weeks ago, which means that it's in > production on quite a large scale, and there are no issues reported > either. Would a patch making it default be accepted ? It's quite handy > given that SMP machines are really pervasive nowadays… Don't forget to turn it off in the MINGW section in the Makefile since we don't have pthreads on Windows. -- Hannes ``` ## Jakub Narebski, 2008-07-10 08:51 Subject: Re: THREADED_DELTA_SEARCH Message-ID: URL: https://gitlist.dev/e/m3hcaydswg.fsf%40localhost.localdomain In-Reply-To: <20080710075327.GD24819@artemis.madism.org> ``` Pierre Habouzit writes: > I'm using it in production for quite a long time now, and I wonder if > there is any specific reason why it's not default. Debian activated it > by default too a couple of weeks ago, which means that it's in > production on quite a large scale, and there are no issues reported > either. Would a patch making it default be accepted ? It's quite handy > given that SMP machines are really pervasive nowadays… Doues our ./configure detects if it makes sense (i.e. if it is SMP machine)? What is the cost of THREADED_DELTA_SEARCH for single processor single core machine? -- Jakub Narebski Poland ShadeHawk on #git ``` ## Pierre Habouzit, 2008-07-10 09:33 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <20080710093308.GF24819@artemis.madism.org> URL: https://gitlist.dev/e/20080710093308.GF24819%40artemis.madism.org In-Reply-To: ``` On Thu, Jul 10, 2008 at 08:51:06AM +0000, Jakub Narebski wrote: > Pierre Habouzit writes: > > > I'm using it in production for quite a long time now, and I wonder if > > there is any specific reason why it's not default. Debian activated it > > by default too a couple of weeks ago, which means that it's in > > production on quite a large scale, and there are no issues reported > > either. Would a patch making it default be accepted ? It's quite handy > > given that SMP machines are really pervasive nowadays… > > Doues our ./configure detects if it makes sense (i.e. if it is SMP > machine)? I don't know if it does, but it wouldn't make sense for distros. > What is the cost of THREADED_DELTA_SEARCH for single processor > single core machine? Threading has to be enabled in the configuration, and there is a --thread=0 mode that enables autodetection of the number of CPUs online. so there is no overhead for single CPU systems. -- ·O· Pierre Habouzit ··O madcoder@debian.org OOO http://www.madism.org ``` ## Junio C Hamano, 2008-07-10 15:34 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <7vy749pxbe.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vy749pxbe.fsf%40gitster.siamese.dyndns.org In-Reply-To: <20080710075327.GD24819@artemis.madism.org> ``` Pierre Habouzit writes: > I'm using it in production for quite a long time now, and I wonder if > there is any specific reason why it's not default. Debian activated it > by default too a couple of weeks ago, which means that it's in > production on quite a large scale, and there are no issues reported > either. Would a patch making it default be accepted ? It's quite handy > given that SMP machines are really pervasive nowadays… There were two issues, IIRC. (1) resulting pack could be suboptimal, due to delta-chain getting cut betweeen the worker thread boundary. (2) exploding memory use, suspected to be due to malloc pool fragmentation under multithreading. The former was only an issue with early iterations of the code and the current code has much improved behaviour on this point, but I do not recall the latter issue has been addressed. ``` ## Nicolas Pitre, 2008-07-10 16:21 Subject: Re: THREADED_DELTA_SEARCH Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.10.0807101212000.12484%40xanadu.home In-Reply-To: <7vy749pxbe.fsf@gitster.siamese.dyndns.org> ``` On Thu, 10 Jul 2008, Junio C Hamano wrote: > Pierre Habouzit writes: > > > I'm using it in production for quite a long time now, and I wonder if > > there is any specific reason why it's not default. Debian activated it > > by default too a couple of weeks ago, which means that it's in > > production on quite a large scale, and there are no issues reported > > either. Would a patch making it default be accepted ? It's quite handy > > given that SMP machines are really pervasive nowadays… > > There were two issues, IIRC. > > (1) resulting pack could be suboptimal, due to delta-chain getting cut > betweeen the worker thread boundary. > > (2) exploding memory use, suspected to be due to malloc pool > fragmentation under multithreading. > > The former was only an issue with early iterations of the code and the > current code has much improved behaviour on this point, I would say it is even negligible. > but I do not > recall the latter issue has been addressed. Well, for "standard" repositories such as the Linux kernel, things always worked just fine. And commit eac12e2d is apparently helping a lot with the remaining odd cases. And if someone has problems due to this then a simple 'git config --global pack.threads 1' would restore the non threaded behavior. Nicolas ``` ## Andreas Ericsson, 2008-07-10 16:50 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <48763DBB.3040708@op5.se> URL: https://gitlist.dev/e/48763DBB.3040708%40op5.se In-Reply-To: <4875C1CF.1010604@viscovery.net> ``` Johannes Sixt wrote: > Pierre Habouzit schrieb: >> I'm using it in production for quite a long time now, and I wonder if >> there is any specific reason why it's not default. Debian activated it >> by default too a couple of weeks ago, which means that it's in >> production on quite a large scale, and there are no issues reported >> either. Would a patch making it default be accepted ? It's quite handy >> given that SMP machines are really pervasive nowadays… > > Don't forget to turn it off in the MINGW section in the Makefile since we > don't have pthreads on Windows. > I started trying that, but ran into problems almost immediately. Enabling pthreads by default and disabling it for certain systems means we screw over those poor sods that have their own config.mak files on systems where there's no to include, or where that header file requires a bunch of other stuff to be in first. On the one hand, that's not nice. On the other hand, those who have a config.mak are probably savvy enough figure out how to amend it disable threads as well. Opinions? I'm all for enabling THREADED_DELTA_SEARCH by default. I've been using it ever since v1.5.4.3-193-g833e3df when using the same .gitconfig on all my systems became a sane thing to do and I haven't seen any ill effects what so ever from it, but how much can we screw the build-system in place today, and what other systems would, by default, require the NO_PTHREADS option in the Makefile? -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 ``` ## Alex Riesen, 2008-07-10 20:13 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <20080710201352.GA3717@blimp.local> URL: https://gitlist.dev/e/20080710201352.GA3717%40blimp.local In-Reply-To: ``` Nicolas Pitre, Thu, Jul 10, 2008 18:21:09 +0200: > On Thu, 10 Jul 2008, Junio C Hamano wrote: > > (2) exploding memory use, suspected to be due to malloc pool > > fragmentation under multithreading. ... > > but I do not > > recall the latter issue has been addressed. > > Well, for "standard" repositories such as the Linux kernel, things > always worked just fine. And commit eac12e2d is apparently helping a > lot with the remaining odd cases. And if someone has problems due to > this then a simple 'git config --global pack.threads 1' would restore > the non threaded behavior. Have you ever seen what happens to Windows XP on memory shortage? It freezes *hard*. Not a good behaviour for _default_ settings. And if someone has no problems with memory, then a simple "git config pack.threads 100" would restore the threaded behaviour for a repository which supposedly will never grow up. ``` ## Nicolas Pitre, 2008-07-10 20:26 Subject: Re: THREADED_DELTA_SEARCH Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.10.0807101625330.12484%40xanadu.home In-Reply-To: <20080710201352.GA3717@blimp.local> ``` On Thu, 10 Jul 2008, Alex Riesen wrote: > Nicolas Pitre, Thu, Jul 10, 2008 18:21:09 +0200: > > On Thu, 10 Jul 2008, Junio C Hamano wrote: > > > (2) exploding memory use, suspected to be due to malloc pool > > > fragmentation under multithreading. > ... > > > but I do not > > > recall the latter issue has been addressed. > > > > Well, for "standard" repositories such as the Linux kernel, things > > always worked just fine. And commit eac12e2d is apparently helping a > > lot with the remaining odd cases. And if someone has problems due to > > this then a simple 'git config --global pack.threads 1' would restore > > the non threaded behavior. > > Have you ever seen what happens to Windows XP on memory shortage? Who said this would be enabled on Windows? Nicolas ``` ## Alex Riesen, 2008-07-10 20:29 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <20080710202919.GB3717@blimp.local> URL: https://gitlist.dev/e/20080710202919.GB3717%40blimp.local In-Reply-To: ``` Nicolas Pitre, Thu, Jul 10, 2008 22:26:03 +0200: > On Thu, 10 Jul 2008, Alex Riesen wrote: > > > > Have you ever seen what happens to Windows XP on memory shortage? > > Who said this would be enabled on Windows? > Windows is just an example (exaggerated one). There are just systems were resources are a problem. ``` ## Nicolas Pitre, 2008-07-10 20:36 Subject: Re: THREADED_DELTA_SEARCH Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.10.0807101633480.12484%40xanadu.home In-Reply-To: <20080710202919.GB3717@blimp.local> ``` On Thu, 10 Jul 2008, Alex Riesen wrote: > Nicolas Pitre, Thu, Jul 10, 2008 22:26:03 +0200: > > On Thu, 10 Jul 2008, Alex Riesen wrote: > > > > > > Have you ever seen what happens to Windows XP on memory shortage? > > > > Who said this would be enabled on Windows? > > > > Windows is just an example (exaggerated one). Don't use that example then. > There are just systems were resources are a problem. Then on those systems you simply have to adjust the appropriate knob. That's why those knobs are available after all. Nicolas ``` ## Alex Riesen, 2008-07-10 22:19 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <20080710221922.GA3665@steel.home> URL: https://gitlist.dev/e/20080710221922.GA3665%40steel.home In-Reply-To: ``` Nicolas Pitre, Thu, Jul 10, 2008 22:36:35 +0200: > > There are just systems were resources are a problem. > > Then on those systems you simply have to adjust the appropriate knob. _after_ they crashed? ``` ## Pierre Habouzit, 2008-07-11 07:41 Subject: Re: THREADED_DELTA_SEARCH Message-ID: <20080711074107.GA7783@artemis.madism.org> URL: https://gitlist.dev/e/20080711074107.GA7783%40artemis.madism.org In-Reply-To: <81b0412b0807102259q1046051bwc513eadb50847675@mail.gmail.com> ``` On Fri, Jul 11, 2008 at 05:59:56AM +0000, Alex Riesen wrote: > 2008/7/11 Nicolas Pitre : > > Furthermore, the number of threads is, by default, equivalent to the > > number of detected CPUs. So if you have a 8-CPU machine, I sure hope > > you have the RAM configuration that goes along with such a system. > > ....and while the threads limit is implemented, the global memory limit is not > (well, frankly, I just don't know. Maybe it is there, and I can't find it). You mean --window-memory= ? I agree this feature should be opt-in. Linux, FreeBSD and Solaris should probably do fine I assume ? I'm quite sure the two first will do, I really know the latter less though. -- ·O· Pierre Habouzit ··O madcoder@debian.org OOO http://www.madism.org ```