[RFC \ WISH] Add -o option to git-rev-list

From: Marco Costalba <mcostalba@gmail.com>
Date: Dec 10, 2006, 11:38 UTC
Message-ID: <e5bfff550612100338ye2ca2a0u1c8f29bbc59c5431@mail.gmail.com>

I have done some tests about reading git-rev-list output with different IPC facilities:

- Native Qt QProcess socket based IPC

- pipe based: popen() and fread()

- redirecting git-rev-list to a file (under tmpfs) and 'block reading'
back the file: read() under Qt QFile class. The file, always in
memory, is deleted at the end of loading.

I have tested with different block sizes and different CPU speed on linux and git trees.

The averaged results on linux tree (about 30MB of data) and CPU set at 1.2GHz are:

- QProcess 6734ms

- pipe and fread() with 64KB blocks 4832ms (38% faster then QProcess)

- temporary file and read() with 64KB blocks 4321ms (10% faster then pipe)

I have not enough knowledge to understand why temporary file is faster then pipe. My guess is, after reading some docs around, fread() uses a C standard I/O buffer, while read() is unbuffered.

To make git-rev-list writing a temporary file I create and run a script more or less like:

git rev-list --header --boundary --parents --topo-order HEAD > /tmp/qgit_135902672.txt

There is also some additional kludge to get git-rev-list pid, so to use in case of a cancel request arrives while loading.

So my request is if it is possible to make git-rev-list write _directly_ to a file, without shell redirection, I would ask if it is possible:

- Add a -o, --output option to git-rev-list to specify output file
instead of stdout

- Use an unbuffered binary block write to be as fastest as possible

- Do *not* flush ever the file, it's only a waste under this scenario.

This will let me to avoid the shell launching crap and probably gain also some more speed.

I understand this could be not exactly a top priority feature for git people, but I would really like to get the best possible interface with the plumbing git and the -o options is also a very common one.

Thanks Marco

P.S: On another thread I explained why I see problematic linking directly against libgit.a I rewrite here for completeness:

Show 19 quoted lines

> I've looked again to Shawn idea (and code) of linking qgit
> against libgit.a but I found these two difficult points:
>
> - traverse_commit_list(&revs, show_commit, show_object) is blocking,
> i.e. the GUI will stop responding for few seconds while traversing the
> list. This is easily and transparently solved by the OS scheduler if
> an external process is used for git-rev-list. To solve this in qgit I
> have two ways: 1) call QEventLoop() once in a while from inside
> show_commit()/ show_object() to process pending events  2) Use a
> separate thread (QThread class). The first idea is not nice, the
> second opens a whole a new set of problems and it's a big amount of
> not trivial new code to add.
>
> -  traverse_commit_list() having an internal state it's not
> re-entrant. git-rev-list it's used to load main view data but also
> file history in another tab, and the two calls _could_ be ran
> concurrently. With external process I simply run two instances of
> DataLoader class and consequently two external git-rev-list processes,
> but If I link against libgit.a that would be a big problem.