git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 08/28] refs: sort ref_dirs lazily

From
mhagger@alum.mit.edu <mhagger@alum.mit.edu>
Date
Oct 28, 2011, 12:28 UTC
Message-ID
<1319804921-17545-9-git-send-email-mhagger@alum.mit.edu>
In-Reply-To
<1319804921-17545-1-git-send-email-mhagger@alum.mit.edu>
From: Michael Haggerty <mhagger@alum.mit.edu>

Sort ref_dirs lazily, when the ordering is needed: for searching via search_ref_dir(), and when iterating over them via do_for_each_ref_in_dir() and do_for_each_ref_in_dirs().

This change means that we never have to sort directories recursively, so change sort_ref_dirs() to not recurse.

NOTE: the dirs can now be sorted as a side-effect of other function
calls.  Therefore, it would be problematic to do something from a
each_ref_fn callback that could provoke the sorting of the directory
that is currently being iterated over.  This is not so likely, because
a directory is always sorted just before being iterated over and thus
can be searched through during the iteration without causing a
re-sort.  But if a callback function would add a reference to a parent
directory of the reference in the iteration, then try to resolve a
reference under that directory, inconsistency could result.

Add a comment in refs.h warning against modifications during iteration.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c |   35 +++++++++++++++--------------------
 refs.h |    7 +++++--
 2 files changed, 20 insertions(+), 22 deletions(-)
diff --git a/refs.c b/refs.c
index 8733a08..733d7a8 100644
--- a/refs.c
+++ b/refs.c
@@ -298,9 +298,14 @@ static struct ref_entry *search_ref_dir(struct ref_dir *dir, const char *refname
 
 	/*
 	 * We need dir to be sorted so that binary search works.
-	 * Calling sort_ref_dir() here is not quite as terribly
-	 * inefficient as it looks, because directories that are
-	 * already sorted are not re-sorted.
+	 * Calling sort_ref_dir() here is not as terribly inefficient
+	 * as it looks.  (1) If the directory is already sorted, it is
+	 * not re-sorted. (2) When adding a reference,
+	 * search_ref_dir() is only called to find the containing
+	 * subdirectories; there is no search of the directory to
+	 * which the reference will be stored.  Thus adding a bunch of
+	 * references one after the other to a single subdirectory
+	 * doesn't require *any* intermediate sorting.
 	 */
 	sort_ref_dir(dir);
 
@@ -406,26 +411,16 @@ static int is_dup_ref(const struct ref_entry *ref1, const struct ref_entry *ref2
 }
 
 /*
- * Sort the entries in dir and its subdirectories (if they are not
- * already sorted).
+ * Sort the entries in dir (if they are not already sorted).  Sort
+ * only dir itself, not its subdirectories.
  */
 static void sort_ref_dir(struct ref_dir *dir)
 {
 	int i, j;
 	struct ref_entry *last = NULL;
 
-	if (dir->sorted == dir->nr) {
-		/*
-		 * This directory is already sorted and de-duped, but
-		 * we still have to sort subdirectories.
-		 */
-		for (i = 0; i < dir->nr; i++) {
-			struct ref_entry *entry = dir->entries[i];
-			if (entry->flag & REF_DIR)
-				sort_ref_dir(&entry->u.subdir);
-		}
-		return;
-	}
+	if (dir->sorted == dir->nr)
+		return; /* This directory is already sorted and de-duped */
 
 	qsort(dir->entries, dir->nr, sizeof(*dir->entries), ref_entry_cmp);
 
@@ -435,7 +430,6 @@ static void sort_ref_dir(struct ref_dir *dir)
 		if (last && is_dup_ref(last, entry)) {
 			free_ref_entry(entry);
 		} else if (entry->flag & REF_DIR) {
-			sort_ref_dir(&entry->u.subdir);
 			dir->entries[i++] = entry;
 			last = NULL;
 		} else {
@@ -472,6 +466,7 @@ static int do_for_each_ref_in_dir(struct ref_dir *dir, int offset,
 				  each_ref_fn fn, int trim, int flags, void *cb_data)
 {
 	int i;
+	sort_ref_dir(dir);
 	for (i = offset; i < dir->nr; i++) {
 		struct ref_entry *entry = dir->entries[i];
 		int retval;
@@ -495,6 +490,8 @@ static int do_for_each_ref_in_dirs(struct ref_dir *dir1,
 	int retval;
 	int i1 = 0, i2 = 0;
 
+	sort_ref_dir(dir1);
+	sort_ref_dir(dir2);
 	while (1) {
 		struct ref_entry *e1, *e2, *entry;
 		int cmp;
@@ -746,7 +743,6 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)
 		    !get_sha1_hex(refline + 1, sha1))
 			hashcpy(last->u.value.peeled, sha1);
 	}
-	sort_ref_dir(dir);
 }
 
 void add_extra_ref(const char *refname, const unsigned char *sha1, int flag)
@@ -849,7 +845,6 @@ static struct ref_dir *get_loose_refs(struct ref_cache *refs)
 {
 	if (!refs->did_loose) {
 		get_ref_dir(refs, "refs", &refs->loose);
-		sort_ref_dir(&refs->loose);
 		refs->did_loose = 1;
 	}
 	return &refs->loose;
diff --git a/refs.h b/refs.h
index d498291..5bb4678 100644
--- a/refs.h
+++ b/refs.h
@@ -15,8 +15,11 @@ struct ref_lock {
 #define REF_ISBROKEN 0x04
 
 /*
- * Calls the specified function for each ref file until it returns nonzero,
- * and returns the value
+ * Calls the specified function for each ref file until it returns
+ * nonzero, and returns the value.  Please note that it is not safe to
+ * modify references while an iteration is in progress, unless the
+ * same callback function invocation that modifies the reference also
+ * returns a nonzero value to immediately stop the iteration.
  */
 typedef int each_ref_fn(const char *refname, const unsigned char *sha1, int flags, void *cb_data);
 extern int head_ref(each_ref_fn, void *);

-- 
1.7.7
Previous: mhagger@alum.mit.eduNext: mhagger@alum.mit.edu
Message 9 of 36 in “Store references hierarchically in cache”
  1. 00/28 Store references hierarchically in cachemhagger@alum.mit.edu, Oct 28, 2011
  2. 01/28 refs.c: reorder definitions more logicallymhagger@alum.mit.edu, Oct 28, 2011
  3. 02/28 free_ref_entry(): new functionmhagger@alum.mit.edu, Oct 28, 2011
  4. 03/28 check_refname_component(): return 0 for zero-length componentsmhagger@alum.mit.edu, Oct 28, 2011
  5. 04/28 struct ref_entry: nest the value part in a unionmhagger@alum.mit.edu, Oct 28, 2011
  6. 05/28 refs.c: rename ref_array -> ref_dirmhagger@alum.mit.edu, Oct 28, 2011
  7. 06/28 refs: store references hierarchicallymhagger@alum.mit.edu, Oct 28, 2011
  8. 07/28 sort_ref_dir(): do not sort if already sortedmhagger@alum.mit.edu, Oct 28, 2011
  9. 08/28 refs: sort ref_dirs lazilymhagger@alum.mit.edu, Oct 28, 2011
  10. 09/28 do_for_each_ref(): only iterate over the subtree that was requestedmhagger@alum.mit.edu, Oct 28, 2011
  11. 10/28 get_ref_dir(): keep track of the current ref_dirmhagger@alum.mit.edu, Oct 28, 2011
  12. 11/28 refs: wrap top-level ref_dirs in ref_entriesmhagger@alum.mit.edu, Oct 28, 2011
  13. 12/28 get_packed_refs(): return (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  14. 13/28 get_loose_refs(): return (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  15. 14/28 is_refname_available(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  16. 15/28 find_ref(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  17. 16/28 read_packed_refs(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  18. 17/28 add_ref(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  19. 18/28 find_containing_direntry(): use (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  20. 19/28 search_ref_dir(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  21. 20/28 add_entry(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  22. 21/28 do_for_each_ref_in_dir*(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  23. 22/28 sort_ref_dir(): take (ref_entry *) instead of (ref_dir *)mhagger@alum.mit.edu, Oct 28, 2011
  24. 23/28 struct ref_dir: store a reference to the enclosing ref_cachemhagger@alum.mit.edu, Oct 28, 2011
  25. 24/28 read_loose_refs(): take a (ref_entry *) as argumentmhagger@alum.mit.edu, Oct 28, 2011
  26. 25/28 refs: read loose references lazilymhagger@alum.mit.edu, Oct 28, 2011
  27. 26/28 is_refname_available(): query only possibly-conflicting referencesmhagger@alum.mit.edu, Oct 28, 2011
  28. Fix "is_refname_available(): query only possibly-conflicting references"mhagger@alum.mit.edu, Nov 15, 2011
  29. Junio C HamanoNov 15, 2011
  30. Michael HaggertyNov 15, 2011
  31. Junio C HamanoNov 15, 2011
  32. 27/28 read_packed_refs(): keep track of the directory being worked inmhagger@alum.mit.edu, Oct 28, 2011
  33. 28/28 repack_without_ref(): call clear_packed_ref_cache()mhagger@alum.mit.edu, Oct 28, 2011
  34. Ramkumar RamachandraOct 28, 2011
  35. Michael HaggertyOct 28, 2011
  36. Michael HaggertyNov 16, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.