Re: Git's database structure

From: Theodore Tso <tytso@mit.edu>
Date: Sep 4, 2007, 21:25 UTC
Message-ID: <20070904212507.GA24434@thunk.org>
In-Reply-To: <9e4733910709041044r71264346n341d178565dd0521@mail.gmail.com>

On Tue, Sep 04, 2007 at 01:44:47PM -0400, Jon Smirl wrote:

Show 5 quoted lines

> The current data store design is not very flexible. Databases solved
> the flexibility problem long ago. I'm just wondering if we should
> steal some good ideas out of the database world and apply them to git.
> Ten years from now we may have 100GB git databases and really wish we
> had more flexible ways of querying them.

Databases solved the flexibility problem, at the cost of performance. And if you use full normalized form in your database scheme, it costs you even more in performance, because of all of the joins that you need in order get the information you need to do, you know, useful work as opposed to database wanking.

If you take a look at the really big databases with super high performance requirements, say like those used to managed airline tickets/reservation/fares, you will find that they are not normalized, and they are not relational; they can't afford to be. And if you take a look at some of git competition that use relational databases to store their SCM data, and take a look at how loooooong they they take to do even basic operations, I would say that the onus is on you to prove that normalization is actually a win in terms of real (not theoretical) advantages, and that it doesn't cause performance to go into the toilet.

I think the fundamental disconnect here is that no one is buying your claim that just because the data design is "more flexible" that this is automatically a good thing in and of itself, and we should even for a moment, "put performance aside".

I also don't think that attempting to force git's data structures into database terms makes sense; it is much closer to an filesystem using an object based store --- and very few people except for folks like Hans Resiers believes that Filesystems and Database should be unified....

						- Ted