git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

From
A Large Angry SCM <gitzilla@gmail.com>
Date
Apr 26, 2009, 18:00 UTC
Message-ID
<49F4A138.6040808@gmail.com>
In-Reply-To
<alpine.DEB.1.00.0904261943070.10279@pacific.mpi-cbg.de>
Johannes Schindelin wrote:
Show 28 quoted lines
> Hi,
> 
> On Sun, 26 Apr 2009, A Large Angry SCM wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Sun, 26 Apr 2009, A Large Angry SCM wrote:
>>>
>>>> Another important criteria was which, both or neither of Git and Hg 
>>>> would actually work and perform well on top of Google Code's 
>>>> underling storage system and except to mention they would be using 
>>>> Bigtable, the report did not discuss this. Git on top of Bigtable 
>>>> will not perform well.
>>> Actually, did we not arrive at the conclusion that it could perform 
>>> well at least with the filesystem layer on top of big table, but even 
>>> better if the big tables stored certain chunks (not really all that 
>>> different from the chunks needed for mirror-sync!)?
>>>
>>> Back when I discussed this with a Googler, it was all too obvious that 
>>> they are not interested (and in the meantime I understand why, see my 
>>> other mail).
>> I don't remember the mirror-sync discussion. But I do remember that when 
>> the discussion turned to implementing a filesystem on top of Bigtable 
>> that would not cause performance problems for Git, my response was that 
>> you'd still be much better off going to GFS directly instead of faking a 
>> filesystem on top of Bigtable without all of the Bigtable limitations.
> 
> Umm, GFS is built on top of Bigtable, no?
Other way around.
Show 8 quoted lines
>> Bigtable _is_ appealing to implement the Git object store on. It's too 
>> bad the latency in Bigtable would make it horribly slow.
> 
> If you store one object per Bigtable, yes.  If you store a few undelta'd 
> objects there, and then use the pack run to optimize those tables, I think 
> it would not be horribly slow.  Of course, you'd need to do exactly the 
> same optimizations necessary for mirror-sync, but I might have mentioned 
> that already ;-)

But now you have to find where you stored those "few undelta'd objects" and then go get the object you're interested in. The only way you can win with that scheme is if you can find groups of objects that are (almost) always accessed together, for all objects (and still not get tripped up by the other limitations of Bigtable).

One method would be to group all of the commit objects into one BT entry and then create a BT entry for each commit that contains all the trees and blobs. This may be fast enough for some operations but would cause the storage requirements to explode.

Previous: Johannes SchindelinNext: James Cloos
Message 20 of 26 in “Google Code: Support for Mercurial and Analysis of Git and Mercurial”
  1. Christian CouderApr 26, 2009
  2. Michael WittenApr 26, 2009
  3. Jakub NarebskiApr 26, 2009
  4. Paolo CiarrocchiApr 26, 2009
  5. Johannes SchindelinApr 26, 2009
  6. Jakub NarebskiApr 26, 2009
  7. Johannes SchindelinApr 26, 2009
  8. Alex BlewittApr 26, 2009
  9. Shawn O. PearceApr 27, 2009
  10. Paolo CiarrocchiApr 26, 2009
  11. Matthias AndreeApr 26, 2009
  12. Jakub NarebskiApr 26, 2009
  13. Matthias AndreeApr 26, 2009
  14. Jakub NarebskiApr 26, 2009
  15. A Large Angry SCMApr 26, 2009
  16. Michael WittenApr 26, 2009
  17. Johannes SchindelinApr 26, 2009
  18. A Large Angry SCMApr 26, 2009
  19. Johannes SchindelinApr 26, 2009
  20. A Large Angry SCMApr 26, 2009
  21. James CloosApr 26, 2009
  22. Johannes SchindelinApr 26, 2009
  23. Michael WittenApr 26, 2009
  24. Miles BaderApr 26, 2009
  25. Shawn O. PearceApr 27, 2009
  26. Mark LodatoApr 30, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.