threads / patch / 7704

patchrefs.c: add a function to sort a ref list, rather then sorting on add

Subject: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

## tl;dr

12 messages between Apr 17, 2007 and Apr 19, 2007. Diffs are folded; open one to read it.

replies: 11people: 4as markdown or json

Julian Phillips· Apr 17, 2007, 01:42 UTC · lore

Rather than sorting the refs list while building it, sort in one go after it is built using a merge sort. This has a large performance boost with large numbers of refs.

Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
---
Having got builtin fetch to the point of generating a correct FETCH_HEAD (for a certain path through the code at least), I revisted the speed issue I brought up a while back with the sorting of refs.
Running fetch (builtin version) on a repo with >9000 refs which is up-to-date, using the old sort-on-add I get (best of 5, warm cache):

real 0m4.351s user 0m4.068s sys 0m0.219s

With this patch the same fetch gives (worst of 5, warm cache):

real 0m2.196s user 0m1.870s sys 0m0.212s

Since this is orthogonal to making fetch a builtin, I don't see that it needs to wait ...
 refs.c |   95 +++++++++++++++++++++++++++++++++++++++++++++++++--------------
 1 files changed, 74 insertions(+), 21 deletions(-)
Show changes to refs.c +74 −21
diff --git a/refs.c b/refs.c
index d2b7b7f..23982fc 100644
--- a/refs.c
+++ b/refs.c
@@ -47,22 +47,7 @@ static struct ref_list *add_ref(const char *name, const unsigned char *sha1,
 				struct ref_list **new_entry)
 {
 	int len;
-	struct ref_list **p = &list, *entry;
-
-	/* Find the place to insert the ref into.. */
-	while ((entry = *p) != NULL) {
-		int cmp = strcmp(entry->name, name);
-		if (cmp > 0)
-			break;
-
-		/* Same as existing entry? */
-		if (!cmp) {
-			if (new_entry)
-				*new_entry = entry;
-			return list;
-		}
-		p = &entry->next;
-	}
+	struct ref_list *entry;
 
 	/* Allocate it and add it in.. */
 	len = strlen(name) + 1;
@@ -71,11 +56,79 @@ static struct ref_list *add_ref(const char *name, const unsigned char *sha1,
 	hashclr(entry->peeled);
 	memcpy(entry->name, name, len);
 	entry->flag = flag;
-	entry->next = *p;
-	*p = entry;
+	entry->next = list;
 	if (new_entry)
 		*new_entry = entry;
-	return list;
+	return entry;
+}
+
+/* merge sort the ref list */
+static struct ref_list *sort_ref_list(struct ref_list *list)
+{
+	int psize, qsize, last_merge_count;
+	struct ref_list *p, *q, *l, *e;
+	struct ref_list *new_list = list;
+	int k = 1;
+	int merge_count = 0;
+
+	if (!list)
+		return list;
+
+	do {
+		last_merge_count = merge_count;
+		merge_count = 0;
+
+		psize = 0;
+
+		p = new_list;
+		q = new_list;
+		new_list = NULL;
+		l = NULL;
+
+		while (p) {
+			merge_count++;
+
+			while (psize < k && q->next) {
+				q = q->next;
+				psize++;
+			}
+			qsize = k;
+
+			while ((psize > 0) || (qsize > 0 && q)) {
+				if (qsize == 0 || !q) {
+					e = p;
+					p = p->next;
+					psize--;
+				} else if (psize == 0) {
+					e = q;
+					q = q->next;
+					qsize--;
+				} else if (strcmp(q->name, p->name) < 0) {
+					e = q;
+					q = q->next;
+					qsize--;
+				} else {
+					e = p;
+					p = p->next;
+					psize--;
+				}
+
+				e->next = NULL;
+
+				if (l)
+					l->next = e;
+				if (!new_list)
+					new_list = e;
+				l = e;
+			}
+
+			p = q;
+		};
+
+		k = k * 2;
+	} while ((last_merge_count != merge_count) || (last_merge_count != 1));
+
+	return new_list;
 }
 
 /*
@@ -142,7 +195,7 @@ static void read_packed_refs(FILE *f, struct cached_refs *cached_refs)
 		    !get_sha1_hex(refline + 1, sha1))
 			hashcpy(last->peeled, sha1);
 	}
-	cached_refs->packed = list;
+	cached_refs->packed = sort_ref_list(list);
 }
 
 static struct ref_list *get_packed_refs(void)
@@ -201,7 +254,7 @@ static struct ref_list *get_ref_dir(const char *base, struct ref_list *list)
 		free(ref);
 		closedir(dir);
 	}
-	return list;
+	return sort_ref_list(list);
 }
 
 static struct ref_list *get_loose_refs(void)
-- 
1.5.1.1
Linus Torvalds· Apr 17, 2007, 16:03 UTC · re: Julian Phillips · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Julian Phillips wrote:
Show 6 quoted lines
>
> Rather than sorting the refs list while building it, sort in one go
> after it is built using a merge sort.  This has a large performance
> boost with large numbers of refs.
> 
> Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>

Looks fine. I think that even your new times are a bit high (over two seconds?) but things are clearly better. Have you looked at what takes so long now?

		Linus
Julian Phillips· Apr 17, 2007, 20:17 UTC · re: Linus Torvalds · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Linus Torvalds wrote:
Show 15 quoted lines
>
>
> On Tue, 17 Apr 2007, Julian Phillips wrote:
>>
>> Rather than sorting the refs list while building it, sort in one go
>> after it is built using a merge sort.  This has a large performance
>> boost with large numbers of refs.
>>
>> Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
>
> Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
>
> Looks fine. I think that even your new times are a bit high (over two
> seconds?) but things are clearly better. Have you looked at what takes so
> long now?

It's the tag auto-following code, I'm calling read_ref to see if I already have that tag - and it appears that doing that a few thousand times takes a while.

If I comment out that one line (so the code will _always_ think I have the tags - but I do have them, so ...) I get:

real 0m0.472s user 0m0.277s sys 0m0.181s

Looks like read_ref is the wrong thing to be using ...
-- 
Julian

  ---
"Life is like a buffet; it's not good but there's plenty of it."
Linus Torvalds· Apr 17, 2007, 20:51 UTC · re: Julian Phillips · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Julian Phillips wrote:
> 
> It's the tag auto-following code, I'm calling read_ref to see if I already
> have that tag - and it appears that doing that a few thousand times takes a
> while.

Heh. I think we should probably call read_refs() just once to read them all (when most of them are packed, that's cheap), and then after that, have some way to just check for a match on the refs we have cached.

		Linus
Julian Phillips· Apr 17, 2007, 22:03 UTC · re: Linus Torvalds · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Linus Torvalds wrote:
Show 11 quoted lines
>
>
> On Tue, 17 Apr 2007, Julian Phillips wrote:
>>
>> It's the tag auto-following code, I'm calling read_ref to see if I already
>> have that tag - and it appears that doing that a few thousand times takes a
>> while.
>
> Heh. I think we should probably call read_refs() just once to read them
> all (when most of them are packed, that's cheap), and then after that,
> have some way to just check for a match on the refs we have cached.

I had a look at the exclude_existing function in show-ref. That uses for_each_ref to build a path_list, and path_list_has_path to do the filtering...

Using that I get (worst of 5, warm cache): real 0m0.526s user 0m0.302s sys 0m0.176s

-- 
Julian

  ---
The descent to Hades is the same from every place.
 		-- Anaxagoras
Junio C Hamano· Apr 17, 2007, 22:00 UTC · re: Linus Torvalds · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

Linus Torvalds <torvalds@linux-foundation.org> writes:
Show 13 quoted lines
> On Tue, 17 Apr 2007, Julian Phillips wrote:
>>
>> Rather than sorting the refs list while building it, sort in one go
>> after it is built using a merge sort.  This has a large performance
>> boost with large numbers of refs.
>> 
>> Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
>
> Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
>
> Looks fine. I think that even your new times are a bit high (over two 
> seconds?) but things are clearly better. Have you looked at what takes so 
> long now? 

I wonder why the loss of "we are replacing the same one" case in the original add_ref() was not compensated for in the new sort_ref_list().

I think we would not call add_ref() to the same list with duplicate names, unless (1) filesystem is grossly corrupt, (2) somebody added a new ref while we are walking (how does readdir() behave in such a case???), or (3) packed-refs file is corrupt.

Julian Phillips· Apr 17, 2007, 22:43 UTC · re: Junio C Hamano · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Junio C Hamano wrote:
> I wonder why the loss of "we are replacing the same one" case in
> the original add_ref() was not compensated for in the new
> sort_ref_list().

That would be because it didn't bite me during testing, and I forgot to double check the code I was removing after I was happy that the sort was producing sane output (it was getting rather late).

Thing is, I'm not sure what to do about it anyway. You could drop the duplicate in the sort, but since add_ref returns a pointer to the added entry (and used to simply return a pointer to the one already in the list) it's possible that you would be dropping something that someone had a pointer to.

Show 5 quoted lines
> I think we would not call add_ref() to the same list with
> duplicate names, unless (1) filesystem is grossly corrupt, (2)
> somebody added a new ref while we are walking (how does
> readdir() behave in such a case???), or (3) packed-refs file is
> corrupt.

This combined with the fact that the old code didn't check that the sha1 was the same suggests to me that this behaviour may actually have been a subtle bug? Perhaps the best thing to do is die if we find two entries with the same name when sorting?

-- 
Julian

  ---
What passes for optimism is most often the effect of an intellectual error.
 		-- Raymond Aron, "The Opium of the Intellectuals"
Junio C Hamano· Apr 18, 2007, 19:36 UTC · re: Julian Phillips · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

Julian Phillips <julian@quantumfyre.co.uk> writes:
Show 12 quoted lines
> On Tue, 17 Apr 2007, Junio C Hamano wrote:
> ...
>> I think we would not call add_ref() to the same list with
>> duplicate names, unless (1) filesystem is grossly corrupt, (2)
>> somebody added a new ref while we are walking (how does
>> readdir() behave in such a case???), or (3) packed-refs file is
>> corrupt.
>
> This combined with the fact that the old code didn't check that the
> sha1 was the same suggests to me that this behaviour may actually have
> been a subtle bug?  Perhaps the best thing to do is die if we find two
> entries with the same name when sorting?

I am not sure what readdir() does if somebody adds a new ref while we are walking the directory; I am hoping we would not get the same thing in duplicates, but I dunno.

I think the most sensible thing to do is to check for duplicates, discarding if the SHA-1 match and otherwise dying.

Julian Phillips· Apr 18, 2007, 21:27 UTC · re: Junio C Hamano · lore

[PATCH] refs.c: drop duplicate entries in sort_ref_list

It shouldn't happen that we read duplicate entries into the same list, but just in case make sort_ref_list drop them. If the SHA1s don't match then die instead, as we have no way of knowing which one is the correct one.

Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
---
On Wed, 18 Apr 2007, Junio C Hamano wrote:
Show 22 quoted lines
> Julian Phillips <julian@quantumfyre.co.uk> writes:
>
>> On Tue, 17 Apr 2007, Junio C Hamano wrote:
>> ...
>>> I think we would not call add_ref() to the same list with
>>> duplicate names, unless (1) filesystem is grossly corrupt, (2)
>>> somebody added a new ref while we are walking (how does
>>> readdir() behave in such a case???), or (3) packed-refs file is
>>> corrupt.
>>
>> This combined with the fact that the old code didn't check that the
>> sha1 was the same suggests to me that this behaviour may actually have
>> been a subtle bug?  Perhaps the best thing to do is die if we find two
>> entries with the same name when sorting?
>
> I am not sure what readdir() does if somebody adds a new ref
> while we are walking the directory; I am hoping we would not get
> the same thing in duplicates, but I dunno.
>
> I think the most sensible thing to do is to check for
> duplicates, discarding if the SHA-1 match and otherwise dying.
>
Something like this (on top of my previous patch)?
 refs.c |   34 ++++++++++++++++++++++++++--------
 1 files changed, 26 insertions(+), 8 deletions(-)
Show changes to refs.c +26 −8
diff --git a/refs.c b/refs.c
index 23982fc..5736a0e 100644
--- a/refs.c
+++ b/refs.c
@@ -65,7 +65,7 @@ static struct ref_list *add_ref(const char *name, const unsigned char *sha1,
 /* merge sort the ref list */
 static struct ref_list *sort_ref_list(struct ref_list *list)
 {
-	int psize, qsize, last_merge_count;
+	int psize, qsize, last_merge_count, cmp;
 	struct ref_list *p, *q, *l, *e;
 	struct ref_list *new_list = list;
 	int k = 1;
@@ -103,14 +103,32 @@ static struct ref_list *sort_ref_list(struct ref_list *list)
 					e = q;
 					q = q->next;
 					qsize--;
-				} else if (strcmp(q->name, p->name) < 0) {
-					e = q;
-					q = q->next;
-					qsize--;
 				} else {
-					e = p;
-					p = p->next;
-					psize--;
+					cmp = strcmp(q->name, p->name);
+					if (cmp < 0) {
+						e = q;
+						q = q->next;
+						qsize--;
+					} else if (cmp > 0) {
+						e = p;
+						p = p->next;
+						psize--;
+					} else {
+						if (hashcmp(q->sha1, p->sha1))
+							die("Duplicated ref, "
+							    "and SHA1s don't "
+							    "match: %s",
+							    q->name);
+						warning("Duplicated ref: %s",
+							q->name);
+						e = q;
+						q = q->next;
+						qsize--;
+						free(e);
+						e = p;
+						p = p->next;
+						psize--;
+					}
 				}
 
 				e->next = NULL;
-- 
1.5.1.1
Johannes Sixt· Apr 19, 2007, 08:19 UTC · re: Julian Phillips · lore

Re: [PATCH] refs.c: drop duplicate entries in sort_ref_list

Julian Phillips wrote:
Show 5 quoted lines
> 
> It shouldn't happen that we read duplicate entries into the same list,
> but just in case make sort_ref_list drop them.  If the SHA1s don't
> match then die instead, as we have no way of knowing which one is the
> correct one.
Clone a random repository that has tags, then

$ rm .git/refs/tags/* $ git fetch origin

now lists each tag twice: for the tag object itself and the commit it points to. Is this related to the problem you are solving here? If so, dying is probably not the best in this situation since the tags are still unique.

(rm .git/refs/tags/* may be insane, but just the other day I wanted to clean up my repo from a number of unneeded branches as well as the tags that they contained - to find out that it is not enough to remove the refs: the objects must go away too, otherwise git fetch auto-follows them and the tags come back even though the branch heads are not there anymore.)

$ git version git version 1.5.1.1.27.g91776

-- Hannes
Julian Phillips· Apr 19, 2007, 08:49 UTC · re: Johannes Sixt · lore

Re: [PATCH] refs.c: drop duplicate entries in sort_ref_list

On Thu, 19 Apr 2007, Johannes Sixt wrote:
Show 16 quoted lines
> Julian Phillips wrote:
>>
>> It shouldn't happen that we read duplicate entries into the same list,
>> but just in case make sort_ref_list drop them.  If the SHA1s don't
>> match then die instead, as we have no way of knowing which one is the
>> correct one.
>
> Clone a random repository that has tags, then
>
> $ rm .git/refs/tags/*
> $ git fetch origin
>
> now lists each tag twice: for the tag object itself and the commit it
> points to. Is this related to the problem you are solving here? If so,
> dying is probably not the best in this situation since the tags are
> still unique.

The sort function is only looking at refs. So if you rm .git/refs/tags/* then you don't have any tag refs (assuming no packed-refs). What it is talking about is having two instances of (e.g.) refs/tags/foo.

In the situation you mention above you still shouldn't see duplicated refs.

-- 
Julian

  ---

Ah, so that's what's been wrong with the little fella.  He misses
casual sex.

 		-- Homer Simpson
 		   Two Dozen and One Greyhounds
Julian Phillips· Apr 17, 2007, 21:01 UTC · re: Julian Phillips · lore

Re: [PATCH] refs.c: add a function to sort a ref list, rather then sorting on add

On Tue, 17 Apr 2007, Julian Phillips wrote:
Show 22 quoted lines
> Rather than sorting the refs list while building it, sort in one go
> after it is built using a merge sort.  This has a large performance
> boost with large numbers of refs.
>
> Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
> ---
>
> Having got builtin fetch to the point of generating a correct FETCH_HEAD (for a certain path through the code at least), I revisted the speed issue I brought up a while back with the sorting of refs.
>
> Running fetch (builtin version) on a repo with >9000 refs which is up-to-date, using the old sort-on-add I get (best of 5, warm cache):
>
> real    0m4.351s
> user    0m4.068s
> sys     0m0.219s
>
> With this patch the same fetch gives (worst of 5, warm cache):
>
> real    0m2.196s
> user    0m1.870s
> sys     0m0.212s
>
> Since this is orthogonal to making fetch a builtin, I don't see that it needs to wait ...

In case anyone is curious, doing the same fetch with master (v1.5.1.1-135-gf948792):

master (best of 5, warm cache): real 0m33.962s user 0m23.992s sys 0m9.986s

master + patch (worst of 5, warm cache): real 0m20.821s user 0m10.390s sys 0m9.799s

-- 
Julian

  ---
NANCY!!  Why is everything RED?!

← back to recent threads