{"thread":{"id":"14391","subject":"THREADED_DELTA_SEARCH","startedAt":"2008-07-10T07:53:27Z","lastAt":"2008-07-11T07:41:08Z","messageCount":13,"participants":["Pierre Habouzit","Johannes Sixt","Jakub Narebski","Junio C Hamano","Nicolas Pitre","Andreas Ericsson","Alex Riesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"82795","messageId":"20080710075327.GD24819@artemis.madism.org","threadId":"14391","inReplyTo":null,"subject":"THREADED_DELTA_SEARCH","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-10T07:53:27Z","receivedAt":"2008-07-10T07:53:27Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"\n I'm using it in production for quite a long time now, and I wonder if\nthere is any specific reason why it's not default. Debian activated it\nby default too a couple of weeks ago, which means that it's in\nproduction on quite a large scale, and there are no issues reported\neither. Would a patch making it default be accepted ? It's quite handy\ngiven that SMP machines are really pervasive nowadays…\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"82796","messageId":"4875C1CF.1010604@viscovery.net","threadId":"14391","inReplyTo":"20080710075327.GD24819@artemis.madism.org","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-07-10T08:01:19Z","receivedAt":"2008-07-10T08:01:19Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Pierre Habouzit schrieb:\n> I'm using it in production for quite a long time now, and I wonder if\n> there is any specific reason why it's not default. Debian activated it\n> by default too a couple of weeks ago, which means that it's in\n> production on quite a large scale, and there are no issues reported\n> either. Would a patch making it default be accepted ? It's quite handy\n> given that SMP machines are really pervasive nowadays…\n\nDon't forget to turn it off in the MINGW section in the Makefile since we\ndon't have pthreads on Windows.\n\n-- Hannes\n"},{"id":"82799","messageId":"m3hcaydswg.fsf@localhost.localdomain","threadId":"14391","inReplyTo":"20080710075327.GD24819@artemis.madism.org","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-07-10T08:51:06Z","receivedAt":"2008-07-10T08:51:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Pierre Habouzit writes:\n\n> I'm using it in production for quite a long time now, and I wonder if\n> there is any specific reason why it's not default. Debian activated it\n> by default too a couple of weeks ago, which means that it's in\n> production on quite a large scale, and there are no issues reported\n> either. Would a patch making it default be accepted ? It's quite handy\n> given that SMP machines are really pervasive nowadays…\n\nDoues our ./configure detects if it makes sense (i.e. if it is SMP\nmachine)? What is the cost of THREADED_DELTA_SEARCH for single processor\nsingle core machine?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"82801","messageId":"20080710093308.GF24819@artemis.madism.org","threadId":"14391","inReplyTo":"m3hcaydswg.fsf@localhost.localdomain","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-10T09:33:09Z","receivedAt":"2008-07-10T09:33:09Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Thu, Jul 10, 2008 at 08:51:06AM +0000, Jakub Narebski wrote:\n> Pierre Habouzit writes:\n> \n> > I'm using it in production for quite a long time now, and I wonder if\n> > there is any specific reason why it's not default. Debian activated it\n> > by default too a couple of weeks ago, which means that it's in\n> > production on quite a large scale, and there are no issues reported\n> > either. Would a patch making it default be accepted ? It's quite handy\n> > given that SMP machines are really pervasive nowadays…\n> \n> Doues our ./configure detects if it makes sense (i.e. if it is SMP\n> machine)?\n\n I don't know if it does, but it wouldn't make sense for distros.\n\n> What is the cost of THREADED_DELTA_SEARCH for single processor\n> single core machine?\n\n Threading has to be enabled in the configuration, and there is a\n--thread=0 mode that enables autodetection of the number of CPUs online.\nso there is no overhead for single CPU systems.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"82820","messageId":"7vy749pxbe.fsf@gitster.siamese.dyndns.org","threadId":"14391","inReplyTo":"20080710075327.GD24819@artemis.madism.org","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-10T15:34:45Z","receivedAt":"2008-07-10T15:34:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Pierre Habouzit writes:\n\n> I'm using it in production for quite a long time now, and I wonder if\n> there is any specific reason why it's not default. Debian activated it\n> by default too a couple of weeks ago, which means that it's in\n> production on quite a large scale, and there are no issues reported\n> either. Would a patch making it default be accepted ? It's quite handy\n> given that SMP machines are really pervasive nowadays…\n\nThere were two issues, IIRC.\n\n (1) resulting pack could be suboptimal, due to delta-chain getting cut\n betweeen the worker thread boundary.\n\n (2) exploding memory use, suspected to be due to malloc pool\n fragmentation under multithreading.\n\nThe former was only an issue with early iterations of the code and the\ncurrent code has much improved behaviour on this point, but I do not\nrecall the latter issue has been addressed.\n"},{"id":"82828","messageId":"alpine.LFD.1.10.0807101212000.12484@xanadu.home","threadId":"14391","inReplyTo":"7vy749pxbe.fsf@gitster.siamese.dyndns.org","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-10T16:21:09Z","receivedAt":"2008-07-10T16:21:09Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 10 Jul 2008, Junio C Hamano wrote:\n\n> Pierre Habouzit writes:\n> \n> > I'm using it in production for quite a long time now, and I wonder if\n> > there is any specific reason why it's not default. Debian activated it\n> > by default too a couple of weeks ago, which means that it's in\n> > production on quite a large scale, and there are no issues reported\n> > either. Would a patch making it default be accepted ? It's quite handy\n> > given that SMP machines are really pervasive nowadays…\n> \n> There were two issues, IIRC.\n> \n> (1) resulting pack could be suboptimal, due to delta-chain getting cut\n> betweeen the worker thread boundary.\n> \n> (2) exploding memory use, suspected to be due to malloc pool\n> fragmentation under multithreading.\n> \n> The former was only an issue with early iterations of the code and the\n> current code has much improved behaviour on this point,\n\nI would say it is even negligible.\n\n> but I do not\n> recall the latter issue has been addressed.\n\nWell, for \"standard\" repositories such as the Linux kernel, things \nalways worked just fine. And commit eac12e2d is apparently helping a \nlot with the remaining odd cases. And if someone has problems due to \nthis then a simple 'git config --global pack.threads 1' would restore \nthe non threaded behavior.\n\n\nNicolas\n"},{"id":"82830","messageId":"48763DBB.3040708@op5.se","threadId":"14391","inReplyTo":"4875C1CF.1010604@viscovery.net","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2008-07-10T16:50:03Z","receivedAt":"2008-07-10T16:50:03Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Johannes Sixt wrote:\n> Pierre Habouzit schrieb:\n>> I'm using it in production for quite a long time now, and I wonder if\n>> there is any specific reason why it's not default. Debian activated it\n>> by default too a couple of weeks ago, which means that it's in\n>> production on quite a large scale, and there are no issues reported\n>> either. Would a patch making it default be accepted ? It's quite handy\n>> given that SMP machines are really pervasive nowadays…\n> \n> Don't forget to turn it off in the MINGW section in the Makefile since we\n> don't have pthreads on Windows.\n> \n\nI started trying that, but ran into problems almost immediately. Enabling\npthreads by default and disabling it for certain systems means we screw\nover those poor sods that have their own config.mak files on systems\nwhere there's no to include, or where that header file\nrequires a bunch of other stuff to be in first.\n\nOn the one hand, that's not nice.\nOn the other hand, those who have a config.mak are probably savvy enough\nfigure out how to amend it disable threads as well.\n\nOpinions? I'm all for enabling THREADED_DELTA_SEARCH by default. I've\nbeen using it ever since v1.5.4.3-193-g833e3df when using the same\n.gitconfig on all my systems became a sane thing to do and I haven't\nseen any ill effects what so ever from it, but how much can we screw\nthe build-system in place today, and what other systems would, by\ndefault, require the NO_PTHREADS option in the Makefile?\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"82844","messageId":"20080710201352.GA3717@blimp.local","threadId":"14391","inReplyTo":"alpine.LFD.1.10.0807101212000.12484@xanadu.home","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-07-10T20:13:52Z","receivedAt":"2008-07-10T20:13:52Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Nicolas Pitre, Thu, Jul 10, 2008 18:21:09 +0200:\n> On Thu, 10 Jul 2008, Junio C Hamano wrote:\n> > (2) exploding memory use, suspected to be due to malloc pool\n> > fragmentation under multithreading.\n...\n> > but I do not\n> > recall the latter issue has been addressed.\n> \n> Well, for \"standard\" repositories such as the Linux kernel, things \n> always worked just fine. And commit eac12e2d is apparently helping a \n> lot with the remaining odd cases. And if someone has problems due to \n> this then a simple 'git config --global pack.threads 1' would restore \n> the non threaded behavior.\n\nHave you ever seen what happens to Windows XP on memory shortage?\nIt freezes *hard*. Not a good behaviour for _default_ settings.\n\nAnd if someone has no problems with memory, then a simple \"git config\npack.threads 100\" would restore the threaded behaviour for a\nrepository which supposedly will never grow up.\n"},{"id":"82846","messageId":"alpine.LFD.1.10.0807101625330.12484@xanadu.home","threadId":"14391","inReplyTo":"20080710201352.GA3717@blimp.local","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-10T20:26:03Z","receivedAt":"2008-07-10T20:26:03Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 10 Jul 2008, Alex Riesen wrote:\n\n> Nicolas Pitre, Thu, Jul 10, 2008 18:21:09 +0200:\n> > On Thu, 10 Jul 2008, Junio C Hamano wrote:\n> > > (2) exploding memory use, suspected to be due to malloc pool\n> > > fragmentation under multithreading.\n> ...\n> > > but I do not\n> > > recall the latter issue has been addressed.\n> > \n> > Well, for \"standard\" repositories such as the Linux kernel, things \n> > always worked just fine. And commit eac12e2d is apparently helping a \n> > lot with the remaining odd cases. And if someone has problems due to \n> > this then a simple 'git config --global pack.threads 1' would restore \n> > the non threaded behavior.\n> \n> Have you ever seen what happens to Windows XP on memory shortage?\n\nWho said this would be enabled on Windows?\n\n\nNicolas\n"},{"id":"82847","messageId":"20080710202919.GB3717@blimp.local","threadId":"14391","inReplyTo":"alpine.LFD.1.10.0807101625330.12484@xanadu.home","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-07-10T20:29:19Z","receivedAt":"2008-07-10T20:29:19Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Nicolas Pitre, Thu, Jul 10, 2008 22:26:03 +0200:\n> On Thu, 10 Jul 2008, Alex Riesen wrote:\n> > \n> > Have you ever seen what happens to Windows XP on memory shortage?\n> \n> Who said this would be enabled on Windows?\n> \n\nWindows is just an example (exaggerated one). There are just systems\nwere resources are a problem.\n"},{"id":"82848","messageId":"alpine.LFD.1.10.0807101633480.12484@xanadu.home","threadId":"14391","inReplyTo":"20080710202919.GB3717@blimp.local","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-10T20:36:35Z","receivedAt":"2008-07-10T20:36:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 10 Jul 2008, Alex Riesen wrote:\n\n> Nicolas Pitre, Thu, Jul 10, 2008 22:26:03 +0200:\n> > On Thu, 10 Jul 2008, Alex Riesen wrote:\n> > > \n> > > Have you ever seen what happens to Windows XP on memory shortage?\n> > \n> > Who said this would be enabled on Windows?\n> > \n> \n> Windows is just an example (exaggerated one).\n\nDon't use that example then.\n\n> There are just systems were resources are a problem.\n\nThen on those systems you simply have to adjust the appropriate knob. \nThat's why those knobs are available after all.\n\n\nNicolas\n"},{"id":"82854","messageId":"20080710221922.GA3665@steel.home","threadId":"14391","inReplyTo":"alpine.LFD.1.10.0807101633480.12484@xanadu.home","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-07-10T22:19:22Z","receivedAt":"2008-07-10T22:19:22Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Nicolas Pitre, Thu, Jul 10, 2008 22:36:35 +0200:\n> > There are just systems were resources are a problem.\n> \n> Then on those systems you simply have to adjust the appropriate knob. \n\n_after_ they crashed?\n"},{"id":"82932","messageId":"20080711074107.GA7783@artemis.madism.org","threadId":"14391","inReplyTo":"81b0412b0807102259q1046051bwc513eadb50847675@mail.gmail.com","subject":"Re: THREADED_DELTA_SEARCH","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-11T07:41:08Z","receivedAt":"2008-07-11T07:41:08Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Fri, Jul 11, 2008 at 05:59:56AM +0000, Alex Riesen wrote:\n> 2008/7/11 Nicolas Pitre :\n> > Furthermore, the number of threads is, by default, equivalent to the\n> > number of detected CPUs. So if you have a 8-CPU machine, I sure hope\n> > you have the RAM configuration that goes along with such a system.\n> \n> ....and while the threads limit is implemented, the global memory limit is not\n> (well, frankly, I just don't know. Maybe it is there, and I can't find it).\n\n You mean --window-memory= ?\n\n I agree this feature should be opt-in. Linux, FreeBSD and Solaris\nshould probably do fine I assume ? I'm quite sure the two first will do,\nI really know the latter less though.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"}]}