git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Fast access git-rev-list output: some OS knowledge required

From
MCMarco Costalba <mcostalba@gmail.com>
Date
Dec 6, 2006, 19:24 UTC
Message-ID
<e5bfff550612061124jcd0d94em47793710866776e7@mail.gmail.com>
I ask help to the list because my knowledge on this is not enough.

Currently qgit uses, socket based, QProcess class to read data from 'git rev-list' when loading the repository at startup.

The time it takes to read, without processing, the whole Linux tree with this approach it's almost _double_ of the time it takes 'git rev-list' to write to a file:

$git rev-list --header --boundary --parents --topo-order HEAD >> tmp.txt
We are talking of about 7s against less then 4s, on my box (warm cache).

So I have a patch to make 'git rev-list' writing into a temporary file and then read it in memory, perhaps it's not the cleaner way, but it's faster, about 1s less.

I have browsed Qt sources and found that QProcess uses internal buffers that are then copied again before to be used by the application. File approach uses a call to read() /fread() buired inside the Qt's QFile class, and no intermediate buffers, so perhaps this could be the reason the second way it's faster.

Anyway there are some issues:
1) File tmp.txt is deleted as soon as read, but this is not enough
sometimes to avoid a costly and wasteful write access to disk by the
OS. What is the easiest, portable way to create a temporary 'in memory
only' file, with no disk access? Or at least delay the HD write access
enough to be able to read and delete the file before the fist block of
tmp.txt is flushed to disk?
2) There is a faster/cleaner (and *safe* ) way to access directly 'git
rev-list' output, something like (just as an example):
$git rev-list --header --boundary --parents --topo-order HEAD >> /dev/mem

Or something similar, possibly _simple_ and _portable_ , so to be able to copy the big amount of 'git rev-list' output just once (about 30MB with current tree).

3) Other suggestions?  ;-)
Thanks
Next: Shawn Pearce
Message 1 of 17 in “Fast access git-rev-list output: some OS knowledge required”
  1. Marco CostalbaDec 6, 2006
  2. Shawn PearceDec 6, 2006
  3. Marco CostalbaDec 6, 2006
  4. Shawn PearceDec 6, 2006
  5. Shawn PearceDec 6, 2006
  6. Marco CostalbaDec 6, 2006
  7. Shawn PearceDec 6, 2006
  8. Andreas EricssonDec 7, 2006
  9. Johannes SchindelinDec 7, 2006
  10. Andreas EricssonDec 7, 2006
  11. Johannes SchindelinDec 7, 2006
  12. Marco CostalbaDec 8, 2006
  13. Michael K. EdwardsDec 8, 2006
  14. Marco CostalbaDec 9, 2006
  15. Johannes SchindelinDec 6, 2006
  16. Linus TorvaldsDec 7, 2006
  17. Marco CostalbaDec 7, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.