QGit: Shrink used memory with custom git log format

> Now instead of --pretty=raw a custom made --pretty=format is given,
> this shrinks loaded data of 30% (17MB less on Linux tree) and gives a
> good speed up when you are low on memory (especially on big repos)
> 
> Next step _would_ be to load log message body on demand (another 50%
> reduction) but this has two drawbacks:
> 
> (1) Text search/filter on log message would be broken
> 
> (2) Slower to browse through revisions because for each revision an
> additional git-rev-list /git-log command should be executed to read
> the body
> 
> The second point is worsted by the fact that it is not possible to
> keep a command running and "open" like as example git-diff-tree
> --stdin and feed with additional revision's sha when needed. Avoiding
> the burden to startup a new process each time to read a new log
> message given an sha would let the answer much more quick especially
> on lesser OS's
> 
> Indeed there is a git-rev-list --stdin option but with different
> behaviour from git-diff-tree --stdin and not suitable for this.

> Marco Costalba <mcostalba@gmail.com> wrote:
> > Now instead of --pretty=raw a custom made --pretty=format is given,
> > this shrinks loaded data of 30% (17MB less on Linux tree) and gives a
> > good speed up when you are low on memory (especially on big repos)
> > 
> > Next step _would_ be to load log message body on demand (another 50%
> > reduction) but this has two drawbacks:
> > 
> > (1) Text search/filter on log message would be broken
> > 
> > (2) Slower to browse through revisions because for each revision an
> > additional git-rev-list /git-log command should be executed to read
> > the body
> > 
> > The second point is worsted by the fact that it is not possible to
> > keep a command running and "open" like as example git-diff-tree
> > --stdin and feed with additional revision's sha when needed. Avoiding
> > the burden to startup a new process each time to read a new log
> > message given an sha would let the answer much more quick especially
> > on lesser OS's
> > 
> > Indeed there is a git-rev-list --stdin option but with different
> > behaviour from git-diff-tree --stdin and not suitable for this.
> 
> There was a proposed patch for git-cat-file that would let you run
> it in a --stdin mode; the git-svn folks wanted this to speed up
> fetching raw objects from the repository.  That may help as you
> could get commit bodies (in raw format - not reencoded format!)
> quite efficiently.
> 
> Otherwise I think what you really want here is a libgit that you can
> link into your process and that can efficiently inflate an object
> on demand for you.  Like the work Luiz was working on this past
> summer for GSOC.  Lots of downsides to that current tree though...
> like die() kills the GUI...

>
> > > Indeed there is a git-rev-list --stdin option but with different
> > > behaviour from git-diff-tree --stdin and not suitable for this.
> >
> > There was a proposed patch for git-cat-file that would let you run
> > it in a --stdin mode; the git-svn folks wanted this to speed up
> > fetching raw objects from the repository.  That may help as you
> > could get commit bodies (in raw format - not reencoded format!)
> > quite efficiently.
>

> On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
> > [...]
> > Otherwise I think what you really want here is a libgit that you can
> > link into your process and that can efficiently inflate an object
> > on demand for you.  Like the work Luiz was working on this past
> > summer for GSOC.  Lots of downsides to that current tree though...
> > like die() kills the GUI...
> 
> But then, die() calls die_routine, which you can override.  And C++ has 
> this funny exception mechanism which just begs to be used here.  The only 
> thing you need to add is a way to flush all singletons like the object 
> array.

> On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:
> > On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
> > > [...]
> > > Otherwise I think what you really want here is a libgit that you can
> > > link into your process and that can efficiently inflate an object
> > > on demand for you.  Like the work Luiz was working on this past
> > > summer for GSOC.  Lots of downsides to that current tree though...
> > > like die() kills the GUI...
> > 
> > But then, die() calls die_routine, which you can override.  And C++ has 
> > this funny exception mechanism which just begs to be used here.  The only 
> > thing you need to add is a way to flush all singletons like the object 
> > array.
> 
> Unfortunately, exceptions won't really work. Why? Because to use 
> exceptions, you need to have an exception-safe code. That is the code 
> needs to free any allocated resources when it's aborted by exception. 
> And git code is not exceptions safe. Given the lack of destructors in C, 
> it means registering all resource allocation in some kind of pool, so 
> they can be freed en masse in case of failure. Than you can also use 
> longjmp for die (for C they really behave the same).

> Hi,
>
> On Tue, 27 Nov 2007, Jan Hudec wrote:
>
>> On Tue, Nov 27, 2007 at 10:48:00 +0000, Johannes Schindelin wrote:
>> > On Mon, 26 Nov 2007, Shawn O. Pearce wrote:
>> > > [...]
>> > > Otherwise I think what you really want here is a libgit that you can
>> > > link into your process and that can efficiently inflate an object
>> > > on demand for you.  Like the work Luiz was working on this past
>> > > summer for GSOC.  Lots of downsides to that current tree though...
>> > > like die() kills the GUI...
>> >
>> > But then, die() calls die_routine, which you can override.  And C++
>> has
>> > this funny exception mechanism which just begs to be used here.  The
>> only
>> > thing you need to add is a way to flush all singletons like the object
>> > array.
>>
>> Unfortunately, exceptions won't really work. Why? Because to use
>> exceptions, you need to have an exception-safe code. That is the code
>> needs to free any allocated resources when it's aborted by exception.
>> And git code is not exceptions safe. Given the lack of destructors in C,
>> it means registering all resource allocation in some kind of pool, so
>> they can be freed en masse in case of failure. Than you can also use
>> longjmp for die (for C they really behave the same).
>
> Sorry, I just assumed that you can read my mind (or alternatively remember
> what I suggested a few months ago, namely to "override" xmalloc(),
> xcalloc(), xrealloc() and xfree() (probably you need to create the
> latter)).