Re-organize "git-rev-list --objects" logic

Linus Torvalds· Sep 15, 2005, 22:14 UTC · lore

The logic to calculate the full object list used to be very inter-twined with the logic that looked up the commits.

For no good reason - it's actually a lot simpler to just do that logic as a separate pass.

This improves performance a bit, and uses slightly less memory in my tests, but more importantly it makes the code simpler to work with and follow what it does.

The performance win is less than I had hoped for, but I get:

Before:

	[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
	13.64user 0.42system 0:14.13elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+47947minor)pagefaults 0swaps
	58945

After:

	[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
	11.80user 0.36system 0:12.16elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+42684minor)pagefaults 0swaps
	58945

ie it improved by 2 seconds, and took a 5000+ fewer pages (hey, that's 20MB out of 174MB to go). And got the same number of objects (in theory, the more expensive one might find some more shared objects to avoid. In practice it obviously doesn't).

I know how to make it use _lots_ less memory, which will probably speed it up. But that's for another time, and I'd prefer to see this go in first.

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
----

diff --git a/rev-list.c b/rev-list.c
--- a/rev-list.c
+++ b/rev-list.c
@@ -231,8 +231,6 @@ static void mark_parents_uninteresting(s
 {
 	struct commit_list *parents = commit->parents;
 
-	if (tree_objects)
-		mark_tree_uninteresting(commit->tree);
 	while (parents) {
 		struct commit *commit = parents->item;
 		commit->object.flags |= UNINTERESTING;
@@ -272,29 +270,6 @@ static int everybody_uninteresting(struc
 			continue;
 		return 0;
 	}
-
-	/*
-	 * Ok, go back and mark all the edge trees uninteresting,
-	 * since otherwise we can have situations where a parent
-	 * that was marked uninteresting (and we never even had
-	 * to look at) had lots of objects that we don't want to
-	 * include.
-	 *
-	 * NOTE! This still doesn't mean that the object list is
-	 * "correct", since we may end up listing objects that
-	 * even older commits (that we don't list) do actually
-	 * reference, but it gets us to a minimal list (or very
-	 * close) in practice.
-	 */
-	if (!tree_objects)
-		return 1;
-
-	while (orig) {
-		struct commit *commit = orig->item;
-		if (!parse_commit(commit) && commit->tree)
-			mark_tree_uninteresting(commit->tree);
-		orig = orig->next;
-	}
 	return 1;
 }
 
@@ -370,6 +345,19 @@ static struct commit_list *find_bisectio
 	return best;
 }
 
+static void mark_edges_uninteresting(struct commit_list *list)
+{
+	for ( ; list; list = list->next) {
+		struct commit_list *parents = list->item->parents;
+
+		for ( ; parents; parents = parents->next) {
+			struct commit *commit = parents->item;
+			if (commit->object.flags & UNINTERESTING)
+				mark_tree_uninteresting(commit->tree);
+		}
+	}
+}
+
 static struct commit_list *limit_list(struct commit_list *list)
 {
 	struct commit_list *newlist = NULL;
@@ -388,6 +376,8 @@ static struct commit_list *limit_list(st
 		}
 		p = &commit_list_insert(commit, p)->next;
 	}
+	if (tree_objects)
+		mark_edges_uninteresting(newlist);
 	if (bisect_list)
 		newlist = find_bisection(newlist);
 	return newlist;

Johannes Schindelin· Sep 15, 2005, 22:27 UTC · re: Linus Torvalds · lore

Re: Re-organize "git-rev-list --objects" logic

Hi,

On Thu, 15 Sep 2005, Linus Torvalds wrote:

> I know how to make it use _lots_ less memory, which will probably speed it 
> up. But that's for another time, and I'd prefer to see this go in first.

Mind to elaborate just a bit?

Linus Torvalds· Sep 16, 2005, 18:11 UTC · re: Johannes Schindelin · lore

Re: Re-organize "git-rev-list --objects" logic

On Fri, 16 Sep 2005, Johannes Schindelin wrote:

Show 7 quoted lines

> 
> On Thu, 15 Sep 2005, Linus Torvalds wrote:
> 
> > I know how to make it use _lots_ less memory, which will probably speed it 
> > up. But that's for another time, and I'd prefer to see this go in first.
> 
> Mind to elaborate just a bit?

Right now we keep track of every single object we parse, and never free it.

Some of it is necessary to keep track of whether a SHA has been seen or not, but: we also keep every single relation list around. They're small, but they are _many_. We can trivially free those after following them. That should get us back another few tens of megabytes easily. It's the parent pointers, and the "struct tree_entry" lists.

Now, some memory usage is very fundamental, and we can't get rid of the "struct object" for each thing we've listed (or each thing we've determined needs to be pruned). We can perhaps shrink it some, but we'll always need to have at a minimum the 20-byte sha1 of every object. So if we list/blacklist a hundred thousand objects, there's no way we can avoid using a couple of megs just to keep track of that fact in order to avoid duplicates.

However, I _think_ the majority of the memory use is in the refs. Even a full kernel tree only has about 95,000 objects, but we have many _many_ more of these relationship links. We've got ~9,000 commits, and pretty much each of them will have a different root tree with 20+ entries in it, and even when most of them end up pointing to the same blobs/trees (which is why we only have 95,000 objects - there's _tons_ of sharing), they'll all end up generating a "tree_entry" thing. So just there, you'll have something like 180,000 small allocations.

And yes, a "tree_entry" is smaller than a "struct object" or "struct tree", but with twice as many "struct tree_entries" as there are objects in _total_, that sure adds up pretty quickly.

And the refs. git-fsck-cache needs to be able to look through the dependency chains for all objects. git-rev-list doesn't. Again, they're small, but they're all over.

So I suspect half the memory allocations is just for these things. Maybe I'm overly optimistic. But I think that even without shrinking the object structures themselves, we should be able to make git-rev-list use a lot less memory.

			Linus