{"thread":{"id":"11001","subject":"QGit: Shrink used memory with custom git log format","startedAt":"2007-11-24T08:14:42Z","lastAt":"2007-11-28T15:53:25Z","messageCount":7,"participants":["Marco Costalba","Shawn O. Pearce","Johannes Schindelin","Jan Hudec","jhud7196@artax.karlin.mff.cuni.cz"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"60801","messageId":"e5bfff550711240014n78f24b46qf012957d92b1a8e1@mail.gmail.com","threadId":"11001","inReplyTo":null,"subject":"QGit: Shrink used memory with custom git log format","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-11-24T08:14:42Z","receivedAt":"2007-11-24T08:14:42Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"Hi all,\n\n I have pushed a patch series to\n\ngit://git.kernel.org/pub/scm/qgit/qgit4.git\n\nthat changes the format of git log used to read data from a git repository.\n\nNow instead of --pretty=raw a custom made --pretty=format is given,\nthis shrinks loaded data of 30% (17MB less on Linux tree) and gives a\ngood speed up when you are low on memory (especially on big repos)\n\nNext step _would_ be to load log message body on demand (another 50%\nreduction) but this has two drawbacks:\n\n(1) Text search/filter on log message would be broken\n\n(2) Slower to browse through revisions because for each revision an\nadditional git-rev-list /git-log command should be executed to read\nthe body\n\nThe second point is worsted by the fact that it is not possible to\nkeep a command running and \"open\" like as example git-diff-tree\n--stdin and feed with additional revision's sha when needed. Avoiding\nthe burden to startup a new process each time to read a new log\nmessage given an sha would let the answer much more quick especially\non lesser OS's\n\nIndeed there is a git-rev-list --stdin option but with different\nbehaviour from git-diff-tree --stdin and not suitable for this.\n\nMarco\n"},{"id":"61018","messageId":"20071127015248.GK14735@spearce.org","threadId":"11001","inReplyTo":"e5bfff550711240014n78f24b46qf012957d92b1a8e1@mail.gmail.com","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-11-27T01:52:48Z","receivedAt":"2007-11-27T01:52:48Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Marco Costalba wrote:\n> Now instead of --pretty=raw a custom made --pretty=format is given,\n> this shrinks loaded data of 30% (17MB less on Linux tree) and gives a\n> good speed up when you are low on memory (especially on big repos)\n> \n> Next step _would_ be to load log message body on demand (another 50%\n> reduction) but this has two drawbacks:\n> \n> (1) Text search/filter on log message would be broken\n> \n> (2) Slower to browse through revisions because for each revision an\n> additional git-rev-list /git-log command should be executed to read\n> the body\n> \n> The second point is worsted by the fact that it is not possible to\n> keep a command running and \"open\" like as example git-diff-tree\n> --stdin and feed with additional revision's sha when needed. Avoiding\n> the burden to startup a new process each time to read a new log\n> message given an sha would let the answer much more quick especially\n> on lesser OS's\n> \n> Indeed there is a git-rev-list --stdin option but with different\n> behaviour from git-diff-tree --stdin and not suitable for this.\n\nThere was a proposed patch for git-cat-file that would let you run\nit in a --stdin mode; the git-svn folks wanted this to speed up\nfetching raw objects from the repository. That may help as you\ncould get commit bodies (in raw format - not reencoded format!)\nquite efficiently.\n\nOtherwise I think what you really want here is a libgit that you can\nlink into your process and that can efficiently inflate an object\non demand for you. Like the work Luiz was working on this past\nsummer for GSOC. Lots of downsides to that current tree though...\nlike die() kills the GUI...\n\n-- \nShawn.\n"},{"id":"61044","messageId":"Pine.LNX.4.64.0711271045430.27959@racer.site","threadId":"11001","inReplyTo":"20071127015248.GK14735@spearce.org","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-11-27T10:48:00Z","receivedAt":"2007-11-27T10:48:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 26 Nov 2007, Shawn O. Pearce wrote:\n\n> Marco Costalba wrote:\n> > Now instead of --pretty=raw a custom made --pretty=format is given,\n> > this shrinks loaded data of 30% (17MB less on Linux tree) and gives a\n> > good speed up when you are low on memory (especially on big repos)\n> > \n> > Next step _would_ be to load log message body on demand (another 50%\n> > reduction) but this has two drawbacks:\n> > \n> > (1) Text search/filter on log message would be broken\n> > \n> > (2) Slower to browse through revisions because for each revision an\n> > additional git-rev-list /git-log command should be executed to read\n> > the body\n> > \n> > The second point is worsted by the fact that it is not possible to\n> > keep a command running and \"open\" like as example git-diff-tree\n> > --stdin and feed with additional revision's sha when needed. Avoiding\n> > the burden to startup a new process each time to read a new log\n> > message given an sha would let the answer much more quick especially\n> > on lesser OS's\n> > \n> > Indeed there is a git-rev-list --stdin option but with different\n> > behaviour from git-diff-tree --stdin and not suitable for this.\n> \n> There was a proposed patch for git-cat-file that would let you run\n> it in a --stdin mode; the git-svn folks wanted this to speed up\n> fetching raw objects from the repository. That may help as you\n> could get commit bodies (in raw format - not reencoded format!)\n> quite efficiently.\n> \n> Otherwise I think what you really want here is a libgit that you can\n> link into your process and that can efficiently inflate an object\n> on demand for you. Like the work Luiz was working on this past\n> summer for GSOC. Lots of downsides to that current tree though...\n> like die() kills the GUI...\n\nBut then, die() calls die_routine, which you can override. And C++ has \nthis funny exception mechanism which just begs to be used here. The only \nthing you need to add is a way to flush all singletons like the object \narray.\n\nCiao,\nDscho\n"},{"id":"61061","messageId":"e5bfff550711270436i712d531bge5072e75285c4a4b@mail.gmail.com","threadId":"11001","inReplyTo":"Pine.LNX.4.64.0711271045430.27959@racer.site","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-11-27T12:36:24Z","receivedAt":"2007-11-27T12:36:24Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Nov 27, 2007 11:48 AM, Johannes Schindelin\n wrote:\n>\n> > > Indeed there is a git-rev-list --stdin option but with different\n> > > behaviour from git-diff-tree --stdin and not suitable for this.\n> >\n> > There was a proposed patch for git-cat-file that would let you run\n> > it in a --stdin mode; the git-svn folks wanted this to speed up\n> > fetching raw objects from the repository. That may help as you\n> > could get commit bodies (in raw format - not reencoded format!)\n> > quite efficiently.\n>\n\nThat would be nice.\n\n> > Otherwise I think what you really want here is a libgit that you can\n> > link into your process and that can efficiently inflate an object\n> > on demand for you.\n\nI would think libgit is overkilling for this.\n\nYou probably would not use libgit to just add a single feature but to\nchange completely the interface with git because the required work is\nheavy both on git side and qgit side (you probably would want to run\nthe libgit linked part in a separated thread to avoid GUI soft locks\nduring slow processing, now, because the executed git command is a\ndifferent process from qgit, the OS scheduler takes care of this 'for\nfree').\n\nMarco\n"},{"id":"61115","messageId":"20071127191915.GB9174@efreet.light.src","threadId":"11001","inReplyTo":"Pine.LNX.4.64.0711271045430.27959@racer.site","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2007-11-27T19:19:15Z","receivedAt":"2007-11-27T19:19:15Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:\n> On Mon, 26 Nov 2007, Shawn O. Pearce wrote:\n> > [...]\n> > Otherwise I think what you really want here is a libgit that you can\n> > link into your process and that can efficiently inflate an object\n> > on demand for you. Like the work Luiz was working on this past\n> > summer for GSOC. Lots of downsides to that current tree though...\n> > like die() kills the GUI...\n> \n> But then, die() calls die_routine, which you can override. And C++ has \n> this funny exception mechanism which just begs to be used here. The only \n> thing you need to add is a way to flush all singletons like the object \n> array.\n\nUnfortunately, exceptions won't really work. Why? Because to use exceptions,\nyou need to have an exception-safe code. That is the code needs to free any\nallocated resources when it's aborted by exception. And git code is not\nexceptions safe. Given the lack of destructors in C, it means registering all\nresource allocation in some kind of pool, so they can be freed en masse in\ncase of failure. Than you can also use longjmp for die (for C they really\nbehave the same).\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"},{"id":"61155","messageId":"Pine.LNX.4.64.0711281200070.27959@racer.site","threadId":"11001","inReplyTo":"20071127191915.GB9174@efreet.light.src","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-11-28T12:01:36Z","receivedAt":"2007-11-28T12:01:36Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Nov 2007, Jan Hudec wrote:\n\n> On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:\n> > On Mon, 26 Nov 2007, Shawn O. Pearce wrote:\n> > > [...]\n> > > Otherwise I think what you really want here is a libgit that you can\n> > > link into your process and that can efficiently inflate an object\n> > > on demand for you. Like the work Luiz was working on this past\n> > > summer for GSOC. Lots of downsides to that current tree though...\n> > > like die() kills the GUI...\n> > \n> > But then, die() calls die_routine, which you can override. And C++ has \n> > this funny exception mechanism which just begs to be used here. The only \n> > thing you need to add is a way to flush all singletons like the object \n> > array.\n> \n> Unfortunately, exceptions won't really work. Why? Because to use \n> exceptions, you need to have an exception-safe code. That is the code \n> needs to free any allocated resources when it's aborted by exception. \n> And git code is not exceptions safe. Given the lack of destructors in C, \n> it means registering all resource allocation in some kind of pool, so \n> they can be freed en masse in case of failure. Than you can also use \n> longjmp for die (for C they really behave the same).\n\nSorry, I just assumed that you can read my mind (or alternatively remember \nwhat I suggested a few months ago, namely to \"override\" xmalloc(), \nxcalloc(), xrealloc() and xfree() (probably you need to create the \nlatter)).\n\nCiao,\nDscho\n"},{"id":"61223","messageId":"57101.194.138.12.144.1196265205.squirrel@artax.karlin.mff.cuni.cz","threadId":"11001","inReplyTo":"Pine.LNX.4.64.0711281200070.27959@racer.site","subject":"Re: QGit: Shrink used memory with custom git log format","fromName":"","fromEmail":"jhud7196@artax.karlin.mff.cuni.cz","sentAt":"2007-11-28T15:53:25Z","receivedAt":"2007-11-28T15:53:25Z","isPatch":false,"sender":{"key":"jhud7196@artax.karlin.mff.cuni.cz","avatar":null},"body":"> Hi,\n>\n> On Tue, 27 Nov 2007, Jan Hudec wrote:\n>\n>> On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:\n>> > On Mon, 26 Nov 2007, Shawn O. Pearce wrote:\n>> > > [...]\n>> > > Otherwise I think what you really want here is a libgit that you can\n>> > > link into your process and that can efficiently inflate an object\n>> > > on demand for you. Like the work Luiz was working on this past\n>> > > summer for GSOC. Lots of downsides to that current tree though...\n>> > > like die() kills the GUI...\n>> >\n>> > But then, die() calls die_routine, which you can override. And C++\n>> has\n>> > this funny exception mechanism which just begs to be used here. The\n>> only\n>> > thing you need to add is a way to flush all singletons like the object\n>> > array.\n>>\n>> Unfortunately, exceptions won't really work. Why? Because to use\n>> exceptions, you need to have an exception-safe code. That is the code\n>> needs to free any allocated resources when it's aborted by exception.\n>> And git code is not exceptions safe. Given the lack of destructors in C,\n>> it means registering all resource allocation in some kind of pool, so\n>> they can be freed en masse in case of failure. Than you can also use\n>> longjmp for die (for C they really behave the same).\n>\n> Sorry, I just assumed that you can read my mind (or alternatively remember\n> what I suggested a few months ago, namely to \"override\" xmalloc(),\n> xcalloc(), xrealloc() and xfree() (probably you need to create the\n> latter)).\n\nThat sounds like the easiest (but not necessarily easy) direction towards\nthe goal. Thread-local or global (I don't think git is currently reentrant\nanyway) would do. Also filehanles would have to be taken care of and\neverything checked for using malloc, calloc, strdup and other libc\nfunctions directly.\n\nThan die could longjmp out to a specified buffer and could be safely\noverriden to throw exception for C++ apps.\n\n--\n - Jan Hudec \n"}]}