Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 29, 2007, 03:22 UTC
Message-ID: <20070529032221.GE7044@spearce.org>
In-Reply-To: <alpine.LFD.0.98.0705282008400.26602@woody.linux-foundation.org>

Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 8 quoted lines

> 
> On Mon, 28 May 2007, Shawn O. Pearce wrote:
> > 
> > What if we use a "slow" storage by "refs/notes/$objname/$notename",
> 
> This _really_ won't scale. Even if the notes don't exist, just doing the 
> lookup (which will fail for most commits) will be horribly slow, and will 
> populate the dentry cache with negative entries.

Yes.

I think you missed what I was trying to say. We *definately* do not want to probe the OS and ask "do you have $objname1? $objname2?" for exactly the reason you just stated. (Though you being some sort of kernel guru means you know a hellva lot more about that dentry cache thing than I do!)

What I meant was something more like:

	DIR *d = opendir("refs/notes");
	if (d) {
		... notes can exist in both packfiles and "loose" ...
		only_packed_notes = 0;

		while (readdir(d)) {
			... the entry name here is the name of an object ...
			... stuff into a hash table, we can come back later ...
			... for the subdirectory contents if we need it ...
		}
		closedir(d);
		... also examine "packed-refs", in case any are there ...
	} else {
		only_packed_notes = 1;
		... we *only* have notes in packfiles, if we have any at all...
	}

Then looking up a note is just a probe into our in-memory hash
(if only_packed_notes is false) and a probe into the packfile(s) to
find the notes for the object.  Not very expensive if the packfiles
have the reverse obj->tag mappings indexed within them.

> To get good filesystem performance, you have to
>  - have reasonable hit-rates (and looking it up for each commit is _not_ 
>    going to do that)
>  - not have millions of objects.

Which is why we:

  a) allow these things to migrate into packed-refs, because
  getting into there is a hellva lot cheaper than getting into
  a packfile;

  b) move them into a packfile when we repack loose objects,
  because then we have really good access.

  c) take them out of packed-refs once they are into a packfile,
  and get them out of the loose refs/notes directory as early as
  possible.

-- 
Shawn.