git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: WARNING! Object DB conversion (was Re: [PATCH] write-tree performance problems)

From
MUMartin Uecker <muecker@gmx.de>
Date
Apr 20, 2005, 13:24 UTC
Message-ID
<20050420132446.GA10126@macavity>
In-Reply-To
<2cfc403205042005116484231c@mail.gmail.com>
On Wed, Apr 20, 2005 at 10:11:10PM +1000, Jon Seymour wrote:
Show 15 quoted lines
> On 4/20/05, Linus Torvalds <torvalds@osdl.org> wrote:
> > 
> > 
> > I converted my git archives (kernel and git itself) to do the SHA1 hash
> > _before_ the compression phase.
> > 
> 
> Linus,
>  
>  Am I correct to understand that with this change, all the objects in
> the database are still being compressed (so no net performance benefit
> now), but by doing the SHA1 calculations before compression you are
> keeping open the possibility that at some point in the future you may
> use a different compression technique (including none at all) for some
> or all of the objects?

The main point is not about trying different compression techniques but that you don't need to compress at all just to calculate the hash of some data. (to know if it is unchanged for example)

There are still some other design decisions I am worried about:

The storage method of the database of a collection of files in the underlying file system. Because of the random nature of the hashes this leads to a horrible amount of seeking for all operations which walk the logical structure of some tree stored in the database.

Why not store all objects linearized in one or more flat file?

The other thing I don't like is the use of a sha1 for a complete file. Switching to some kind of hash tree would allow to introduce chunks later. This has two advantages:

It would allow git to scale to repositories of large binary files. And it would allow to build a very cool content transport algorithm for those repositories. This algorithm could combine all the advantages of bittorrent and rsync (without the cpu load).

And it would allow trivial merging of patches which apply to different chunks of a file in exact the same way as merging changesets which apply to different files in a tree.

Martin
-- 
One night, when little Giana from Milano was fast asleep,
she had a strange dream.
Previous: Jon SeymourNext: Morten Welinder
Message 14 of 54 in “write-tree performance problems”
  1. write-tree performance problemsChris Mason, Apr 19, 2005
  2. Linus TorvaldsApr 19, 2005
  3. Chris MasonApr 19, 2005
  4. Linus TorvaldsApr 19, 2005
  5. Chris MasonApr 19, 2005
  6. Linus TorvaldsApr 19, 2005
  7. Chris MasonApr 20, 2005
  8. Linus TorvaldsApr 20, 2005
  9. Linus TorvaldsApr 20, 2005
  10. H. Peter AnvinApr 20, 2005
  11. WARNING! Object DB conversion (was Re: [PATCH] write-tree performance problems)Linus Torvalds, Apr 20, 2005
  12. Ingo MolnarApr 20, 2005
  13. Jon SeymourApr 20, 2005
  14. Martin UeckerApr 20, 2005
  15. Morten WelinderApr 20, 2005
  16. Jon SeymourApr 20, 2005
  17. C. Scott AnanianApr 20, 2005
  18. Martin UeckerApr 20, 2005
  19. C. Scott AnanianApr 20, 2005
  20. Martin UeckerApr 20, 2005
  21. Martin UeckerApr 20, 2005
  22. Blob chunking code. [First look.]C. Scott Ananian, Apr 20, 2005
  23. Blob chunking code. [Second look]C. Scott Ananian, Apr 20, 2005
  24. David WoodhouseApr 20, 2005
  25. Linus TorvaldsApr 20, 2005
  26. David WoodhouseApr 20, 2005
  27. Chris MasonApr 20, 2005
  28. C. Scott AnanianApr 20, 2005
  29. Linus TorvaldsApr 20, 2005
  30. C. Scott AnanianApr 20, 2005
  31. Linus TorvaldsApr 20, 2005
  32. Linus TorvaldsApr 20, 2005
  33. David WillmoreApr 20, 2005
  34. Linus TorvaldsApr 20, 2005
  35. Linus TorvaldsApr 20, 2005
  36. Chris MasonApr 20, 2005
  37. Linus TorvaldsApr 20, 2005
  38. Chris MasonApr 20, 2005
  39. Linus TorvaldsApr 20, 2005
  40. Chris MasonApr 20, 2005
  41. Linus TorvaldsApr 20, 2005
  42. Linus TorvaldsApr 20, 2005
  43. David S. MillerApr 20, 2005
  44. David LangApr 19, 2005
  45. Linus TorvaldsApr 19, 2005
  46. David LangApr 19, 2005
  47. Linus TorvaldsApr 19, 2005
  48. David LangApr 19, 2005
  49. Linus TorvaldsApr 19, 2005
  50. Christopher LiApr 19, 2005
  51. Olivier GalibertApr 19, 2005
  52. C. Scott AnanianApr 19, 2005
  53. Linus TorvaldsApr 20, 2005
  54. C. Scott AnanianApr 20, 2005

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.