threads / patch / 34110

patch, 12 partsFix some reference-related races

Subject: [PATCH 00/12] Fix some reference-related races

## tl;dr

22 messages between Jun 11, 2013 and Jun 19, 2013. Diffs are folded; open one to read it.

replies: 21people: 3as markdown or json

Michael Haggerty· Jun 11, 2013, 21:48 UTC · lore
*This patch series must be built on top of mh/reflife.*

This patch series fixes some races reading loose and packed refs. Most of the problems, and some of the solutions, were pointed out by Jeff King [1] but some other work was necessary to prevent his fixes from causing problems elsewhere.

The basic race being addressed is that at any time "pack-refs --prune" might be run at any time. It rewrites the packed-refs file and then deletes the just-packed loose refs.

Readers, then, to get a self-consistent snapshot of references [2], must be sure to read all of the loose references it will need *before* reading the packed references (or at least verifying that the packed references that it read earlier are still valid). But given the lazy-loading of the loose references cache, this was not always the case.

So the core of this patch series is to force loose references for an iteration to be read all at once into the cache, and *then* to verify that the packed-refs cache is up-to-date and if not to reload it. Similarly, when looking up single references, a loose reference is sought first, and then the validity of the packed-refs cache is verified, and then the loose reference is sought in the cache.

The problem is that there was a lot of code that assumed that the lifetime of the reference cache was essentially infinite. The mh/reflife patch series (which has made it to next) fixed callers who retained pointers to refnames in the cache.

The other problem was that the for_each_ref() functions will die if the ref cache that they are iterating over is freed out from under them. This problem is solved by using reference counts to avoid freeing the old packed ref cache (even if it is no longer valid) until all users are done with it.

Once those are done, it is possible to invalidate the packed refs cache when needed. So (1) we always read all loose references that will be needed in an iteration before the iteration starts, and (2) we add a check (based on file metadata) whenever the packed-refs cache is accessed that it is still up-to-date WRT the packed-refs file, and if not reread it (but leave the old copy in memory as long as its refcount is nonzero).

Along the way, this patch series adds simple transactions around the packed-refs file/cache. The transaction interface is public. I think this is a step in a good direction, because other race conditions not addressed by this patch series are likely to require transactions across the whole reference namespace to be made 100% reliable.

As a stress test, the test suite can be run with a simulated "hyperactive repository" in which the packed-refs file is made to look like it changes every time it is checked (except when its lock is held):

    ------------------------------------ refs.c ------------------------------------
    @@ -1075,8 +1075,8 @@ static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)
     	else
     		packed_refs_file = git_path("packed-refs");
    -	if (refs->packed &&
    -	    !stat_validity_check(&refs->packed->validity, packed_refs_file))
    +	if (refs->packed && !refs->packed->lock
    +	    /*!stat_validity_check(&refs->packed->validity, packed_refs_file)*/)
     		clear_packed_ref_cache(refs);
     	if (!refs->packed) {
It passes the stress test.
[1] http://thread.gmane.org/gmane.comp.version-control.git/223299/focus=223526
Jeff King (2):
  get_packed_ref_cache: reload packed-refs file when it changes
  for_each_ref: load all loose refs before packed refs
Michael Haggerty (10):
  repack_without_ref(): split list curation and entry writing
  pack_refs(): split creation of packed refs and entry writing
  refs: wrap the packed refs cache in a level of indirection
  refs: implement simple transactions for the packed-refs file
  refs: manage lifetime of packed refs cache via reference counting
  do_for_each_entry(): increment the packed refs cache refcount
  packed_ref_cache: increment refcount when locked
  Extract a struct stat_data from cache_entry
  add a stat_validity struct
  refs: do not invalidate the packed-refs cache unnecessarily
 builtin/clone.c    |   7 +-
 builtin/ls-files.c |  12 ++-
 cache.h            |  60 +++++++++--
 read-cache.c       | 181 +++++++++++++++++++------------
 refs.c             | 308 ++++++++++++++++++++++++++++++++++++++++++++---------
 refs.h             |  27 ++++-
 6 files changed, 464 insertions(+), 131 deletions(-)
-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 01/12] repack_without_ref(): split list curation and entry writing

Split repack_without_ref() into multiple passes:
* collect the list of refnames that should be deleted from packed_refs
* delete those refnames from the cache
* write the remainder to the packed-refs file

The purpose of this change is to make the "write the remainder" part reusable.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 48 +++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 11 deletions(-)
Show changes to refs.c +37 −11
diff --git a/refs.c b/refs.c
index d17931a..dccfe14 100644
--- a/refs.c
+++ b/refs.c
@@ -1994,6 +1994,23 @@ static void write_packed_entry(int fd, char *refname, unsigned char *sha1,
 	}
 }
 
+/*
+ * An each_ref_entry_fn that writes the entry to a packed-refs file.
+ */
+static int write_packed_entry_fn(struct ref_entry *entry, void *cb_data)
+{
+	int *fd = cb_data;
+	enum peel_status peel_status = peel_entry(entry, 0);
+
+	if (peel_status != PEEL_PEELED && peel_status != PEEL_NON_TAG)
+		error("internal error: %s is not a valid packed reference!",
+		      entry->name);
+	write_packed_entry(*fd, entry->name, entry->u.value.sha1,
+			   peel_status == PEEL_PEELED ?
+			   entry->u.value.peeled : NULL);
+	return 0;
+}
+
 struct ref_to_prune {
 	struct ref_to_prune *next;
 	unsigned char sha1[20];
@@ -2113,14 +2130,18 @@ int pack_refs(unsigned int flags)
 	return 0;
 }
 
-static int repack_ref_fn(struct ref_entry *entry, void *cb_data)
+/*
+ * If entry should be deleted from packed-refs, add it to the string
+ * list pointed to by cb_data.
+ */
+static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
 {
-	int *fd = cb_data;
-	enum peel_status peel_status;
+	struct string_list *refs_to_delete = cb_data;
 
 	if (entry->flag & REF_ISBROKEN) {
 		/* This shouldn't happen to packed refs. */
 		error("%s is broken!", entry->name);
+		string_list_append(refs_to_delete, entry->name);
 		return 0;
 	}
 	if (!has_sha1_file(entry->u.value.sha1)) {
@@ -2130,7 +2151,7 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)
 		if (read_ref_full(entry->name, sha1, 0, &flags))
 			/* We should at least have found the packed ref. */
 			die("Internal error");
-		if ((flags & REF_ISSYMREF) || !(flags & REF_ISPACKED))
+		if ((flags & REF_ISSYMREF) || !(flags & REF_ISPACKED)) {
 			/*
 			 * This packed reference is overridden by a
 			 * loose reference, so it is OK that its value
@@ -2139,9 +2160,11 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)
 			 * collected.  For this purpose we don't even
 			 * care whether the loose reference itself is
 			 * invalid, broken, symbolic, etc.  Silently
-			 * omit the packed reference from the output.
+			 * remove the packed reference.
 			 */
+			string_list_append(refs_to_delete, entry->name);
 			return 0;
+		}
 		/*
 		 * There is no overriding loose reference, so the fact
 		 * that this reference doesn't refer to a valid object
@@ -2150,14 +2173,10 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)
 		 * the output.
 		 */
 		error("%s does not point to a valid object!", entry->name);
+		string_list_append(refs_to_delete, entry->name);
 		return 0;
 	}
 
-	peel_status = peel_entry(entry, 0);
-	write_packed_entry(*fd, entry->name, entry->u.value.sha1,
-			   peel_status == PEEL_PEELED ?
-			   entry->u.value.peeled : NULL);
-
 	return 0;
 }
 
@@ -2165,6 +2184,8 @@ static int repack_without_ref(const char *refname)
 {
 	int fd;
 	struct ref_dir *packed;
+	struct string_list refs_to_delete = STRING_LIST_INIT_DUP;
+	struct string_list_item *ref_to_delete;
 
 	if (!get_packed_ref(refname))
 		return 0; /* refname does not exist in packed refs */
@@ -2185,8 +2206,13 @@ static int repack_without_ref(const char *refname)
 		rollback_lock_file(&packlock);
 		return 0;
 	}
+	do_for_each_entry_in_dir(packed, 0, curate_packed_ref_fn, &refs_to_delete);
+	for_each_string_list_item(ref_to_delete, &refs_to_delete) {
+		if (remove_entry(packed, ref_to_delete->string) == -1)
+			die("internal error");
+	}
 	write_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
-	do_for_each_entry_in_dir(packed, 0, repack_ref_fn, &fd);
+	do_for_each_entry_in_dir(packed, 0, write_packed_entry_fn, &fd);
 	return commit_lock_file(&packlock);
 }
 

-- 
1.8.3
Jeff King· Jun 12, 2013, 11:38 UTC · re: Michael Haggerty · lore

Re: [PATCH 01/12] repack_without_ref(): split list curation and entry writing

On Tue, Jun 11, 2013 at 11:48:21PM +0200, Michael Haggerty wrote:
Show 10 quoted lines
> Split repack_without_ref() into multiple passes:
> 
> * collect the list of refnames that should be deleted from packed_refs
> 
> * delete those refnames from the cache
> 
> * write the remainder to the packed-refs file
> 
> The purpose of this change is to make the "write the remainder" part
> reusable.

It took me several reads to figure out what was going on here, because I did not see the deleted ref passed to the list of items to delete from packed_refs. The part I was missing is something like:

  The repack_without_ref() function first removes the deleted ref from
  the internal packed-refs list, then writes the packed-refs list to
  disk, omitting any broken or stale entries. This patch splits that
  second step into multiple passes:
     ...
Is that accurate?
-Peff
Michael Haggerty· Jun 12, 2013, 11:56 UTC · re: Jeff King · lore

Re: [PATCH 01/12] repack_without_ref(): split list curation and entry writing

On 06/12/2013 01:38 PM, Jeff King wrote:
Show 25 quoted lines
> On Tue, Jun 11, 2013 at 11:48:21PM +0200, Michael Haggerty wrote:
> 
>> Split repack_without_ref() into multiple passes:
>>
>> * collect the list of refnames that should be deleted from packed_refs
>>
>> * delete those refnames from the cache
>>
>> * write the remainder to the packed-refs file
>>
>> The purpose of this change is to make the "write the remainder" part
>> reusable.
> 
> It took me several reads to figure out what was going on here, because I
> did not see the deleted ref passed to the list of items to delete from
> packed_refs. The part I was missing is something like:
> 
>   The repack_without_ref() function first removes the deleted ref from
>   the internal packed-refs list, then writes the packed-refs list to
>   disk, omitting any broken or stale entries. This patch splits that
>   second step into multiple passes:
> 
>      ...
> 
> Is that accurate?
Yes.  You are right that I should make that clearer.

Thanks, Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 02/12] pack_refs(): split creation of packed refs and entry writing

Split pack_refs() into multiple passes:
* Iterate over loose refs.  For each one that can be turned into a
  packed ref, create a corresponding entry in the packed refs cache.
* Write the packed refs to the packed-refs file.

This change isolates the mutation of the packed-refs file to a single place.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 48 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 34 insertions(+), 14 deletions(-)
Show changes to refs.c +34 −14
diff --git a/refs.c b/refs.c
index dccfe14..bb3640f 100644
--- a/refs.c
+++ b/refs.c
@@ -2019,35 +2019,50 @@ struct ref_to_prune {
 
 struct pack_refs_cb_data {
 	unsigned int flags;
+	struct ref_dir *packed_refs;
 	struct ref_to_prune *ref_to_prune;
-	int fd;
 };
 
-static int pack_one_ref(struct ref_entry *entry, void *cb_data)
+/*
+ * An each_ref_entry_fn that is run over loose references only.  If
+ * the loose reference can be packed, add an entry in the packed ref
+ * cache.  If the reference should be pruned, also add it to
+ * ref_to_prune in the pack_refs_cb_data.
+ */
+static int pack_if_possible_fn(struct ref_entry *entry, void *cb_data)
 {
 	struct pack_refs_cb_data *cb = cb_data;
 	enum peel_status peel_status;
+	struct ref_entry *packed_entry;
 	int is_tag_ref = !prefixcmp(entry->name, "refs/tags/");
 
-	/* ALWAYS pack refs that were already packed or are tags */
-	if (!(cb->flags & PACK_REFS_ALL) && !is_tag_ref &&
-	    !(entry->flag & REF_ISPACKED))
+	/* ALWAYS pack tags */
+	if (!(cb->flags & PACK_REFS_ALL) && !is_tag_ref)
 		return 0;
 
 	/* Do not pack symbolic or broken refs: */
 	if ((entry->flag & REF_ISSYMREF) || !ref_resolves_to_object(entry))
 		return 0;
 
+	/* Add a packed ref cache entry equivalent to the loose entry. */
 	peel_status = peel_entry(entry, 1);
 	if (peel_status != PEEL_PEELED && peel_status != PEEL_NON_TAG)
 		die("internal error peeling reference %s (%s)",
 		    entry->name, sha1_to_hex(entry->u.value.sha1));
-	write_packed_entry(cb->fd, entry->name, entry->u.value.sha1,
-			   peel_status == PEEL_PEELED ?
-			   entry->u.value.peeled : NULL);
+	packed_entry = find_ref(cb->packed_refs, entry->name);
+	if (packed_entry) {
+		/* Overwrite existing packed entry with info from loose entry */
+		packed_entry->flag = REF_ISPACKED | REF_KNOWS_PEELED;
+		hashcpy(packed_entry->u.value.sha1, entry->u.value.sha1);
+	} else {
+		packed_entry = create_ref_entry(entry->name, entry->u.value.sha1,
+						REF_ISPACKED | REF_KNOWS_PEELED, 0);
+		add_ref(cb->packed_refs, packed_entry);
+	}
+	hashcpy(packed_entry->u.value.peeled, entry->u.value.peeled);
 
-	/* If the ref was already packed, there is no need to prune it. */
-	if ((cb->flags & PACK_REFS_PRUNE) && !(entry->flag & REF_ISPACKED)) {
+	/* Schedule the loose reference for pruning if requested. */
+	if ((cb->flags & PACK_REFS_PRUNE)) {
 		int namelen = strlen(entry->name) + 1;
 		struct ref_to_prune *n = xcalloc(1, sizeof(*n) + namelen);
 		hashcpy(n->sha1, entry->u.value.sha1);
@@ -2114,16 +2129,21 @@ static struct lock_file packlock;
 int pack_refs(unsigned int flags)
 {
 	struct pack_refs_cb_data cbdata;
+	int fd;
 
 	memset(&cbdata, 0, sizeof(cbdata));
 	cbdata.flags = flags;
 
-	cbdata.fd = hold_lock_file_for_update(&packlock, git_path("packed-refs"),
-					      LOCK_DIE_ON_ERROR);
+	fd = hold_lock_file_for_update(&packlock, git_path("packed-refs"),
+				       LOCK_DIE_ON_ERROR);
+	cbdata.packed_refs = get_packed_refs(&ref_cache);
 
-	write_or_die(cbdata.fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
+	do_for_each_entry_in_dir(get_loose_refs(&ref_cache), 0,
+				 pack_if_possible_fn, &cbdata);
+
+	write_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
+	do_for_each_entry_in_dir(cbdata.packed_refs, 0, write_packed_entry_fn, &fd);
 
-	do_for_each_entry(&ref_cache, "", pack_one_ref, &cbdata);
 	if (commit_lock_file(&packlock) < 0)
 		die_errno("unable to overwrite old ref-pack file");
 	prune_refs(cbdata.ref_to_prune);

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 03/12] refs: wrap the packed refs cache in a level of indirection

As we know, we can solve any problem in this manner. In this case, the problem is to avoid freeing a packed refs cache while somebody is using it. So add a level of indirection as a prelude to reference-counting the packed refs cache.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)
Show changes to refs.c +26 −6
diff --git a/refs.c b/refs.c
index bb3640f..4cee36b 100644
--- a/refs.c
+++ b/refs.c
@@ -802,6 +802,10 @@ static int is_refname_available(const char *refname, const char *oldrefname,
 	return 1;
 }
 
+struct packed_ref_cache {
+	struct ref_entry *root;
+};
+
 /*
  * Future: need to be in "struct repository"
  * when doing a full libification.
@@ -809,7 +813,7 @@ static int is_refname_available(const char *refname, const char *oldrefname,
 static struct ref_cache {
 	struct ref_cache *next;
 	struct ref_entry *loose;
-	struct ref_entry *packed;
+	struct packed_ref_cache *packed;
 	/*
 	 * The submodule name, or "" for the main repo.  We allocate
 	 * length 1 rather than FLEX_ARRAY so that the main ref_cache
@@ -821,7 +825,8 @@ static struct ref_cache {
 static void clear_packed_ref_cache(struct ref_cache *refs)
 {
 	if (refs->packed) {
-		free_ref_entry(refs->packed);
+		free_ref_entry(refs->packed->root);
+		free(refs->packed);
 		refs->packed = NULL;
 	}
 }
@@ -992,24 +997,39 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)
 	}
 }
 
-static struct ref_dir *get_packed_refs(struct ref_cache *refs)
+/*
+ * Get the packed_ref_cache for the specified ref_cache, creating it
+ * if necessary.
+ */
+static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)
 {
 	if (!refs->packed) {
 		const char *packed_refs_file;
 		FILE *f;
 
-		refs->packed = create_dir_entry(refs, "", 0, 0);
+		refs->packed = xcalloc(1, sizeof(*refs->packed));
+		refs->packed->root = create_dir_entry(refs, "", 0, 0);
 		if (*refs->name)
 			packed_refs_file = git_path_submodule(refs->name, "packed-refs");
 		else
 			packed_refs_file = git_path("packed-refs");
 		f = fopen(packed_refs_file, "r");
 		if (f) {
-			read_packed_refs(f, get_ref_dir(refs->packed));
+			read_packed_refs(f, get_ref_dir(refs->packed->root));
 			fclose(f);
 		}
 	}
-	return get_ref_dir(refs->packed);
+	return refs->packed;
+}
+
+static struct ref_dir *get_packed_ref_dir(struct packed_ref_cache *packed_ref_cache)
+{
+	return get_ref_dir(packed_ref_cache->root);
+}
+
+static struct ref_dir *get_packed_refs(struct ref_cache *refs)
+{
+	return get_packed_ref_dir(get_packed_ref_cache(refs));
 }
 
 void add_packed_ref(const char *refname, const unsigned char *sha1)

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 04/12] refs: implement simple transactions for the packed-refs file

Handle simple transactions for the packed-refs file at the packed_ref_cache level via new functions lock_packed_refs(), commit_packed_refs(), and rollback_packed_refs().

Only allow the packed ref cache to be modified (via add_packed_ref()) while the packed refs file is locked.

Change clone to add the new references within a transaction.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
The API docs are not clear about whether it is kosher to read
lock_file::fd directly.  It is only done in one file outside of
lockfile.c.  So this patch stores the fd of the lockfile separately in
struct packed_ref_cache, even though the same struct also has a
pointer to the struct lock_file.

So please let me know if it is OK to read lock_file::fd directly. If so, then I will drop the fd member of struct packed_ref_cache, as well as the local variable "fd" in lock_packed_refs().

 builtin/clone.c |  7 ++++-
 refs.c          | 91 +++++++++++++++++++++++++++++++++++++++++++++++----------
 refs.h          | 27 +++++++++++++++--
 3 files changed, 106 insertions(+), 19 deletions(-)
Show changes to 3 files +106 −19

builtin/clone.c, refs.c, refs.h

diff --git a/builtin/clone.c b/builtin/clone.c
index 66bff57..b0c000a 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -489,17 +489,22 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	return local_refs;
 }
 
+static struct lock_file packed_refs_lock;
+
 static void write_remote_refs(const struct ref *local_refs)
 {
 	const struct ref *r;
 
+	lock_packed_refs(&packed_refs_lock, LOCK_DIE_ON_ERROR);
+
 	for (r = local_refs; r; r = r->next) {
 		if (!r->peer_ref)
 			continue;
 		add_packed_ref(r->peer_ref->name, r->old_sha1);
 	}
 
-	pack_refs(PACK_REFS_ALL);
+	if (commit_packed_refs())
+		die_errno("unable to overwrite old ref-pack file");
 }
 
 static void write_followtags(const struct ref *refs, const char *msg)
diff --git a/refs.c b/refs.c
index 4cee36b..114bd5b 100644
--- a/refs.c
+++ b/refs.c
@@ -804,6 +804,16 @@ static int is_refname_available(const char *refname, const char *oldrefname,
 
 struct packed_ref_cache {
 	struct ref_entry *root;
+
+	/*
+	 * Iff the packed-refs file associated with this instance is
+	 * currently locked for writing, this points at the associated
+	 * lock (which is owned by somebody else).
+	 */
+	struct lock_file *lock;
+
+	/* If locked, the file descriptor of the lock file. */
+	int fd;
 };
 
 /*
@@ -825,6 +835,8 @@ static struct ref_cache {
 static void clear_packed_ref_cache(struct ref_cache *refs)
 {
 	if (refs->packed) {
+		if (refs->packed->lock)
+			die("internal error: packed-ref cache cleared while locked");
 		free_ref_entry(refs->packed->root);
 		free(refs->packed);
 		refs->packed = NULL;
@@ -1009,6 +1021,7 @@ static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)
 
 		refs->packed = xcalloc(1, sizeof(*refs->packed));
 		refs->packed->root = create_dir_entry(refs, "", 0, 0);
+		refs->packed->fd = -1;
 		if (*refs->name)
 			packed_refs_file = git_path_submodule(refs->name, "packed-refs");
 		else
@@ -1034,7 +1047,12 @@ static struct ref_dir *get_packed_refs(struct ref_cache *refs)
 
 void add_packed_ref(const char *refname, const unsigned char *sha1)
 {
-	add_ref(get_packed_refs(&ref_cache),
+	struct packed_ref_cache *packed_ref_cache =
+		get_packed_ref_cache(&ref_cache);
+
+	if (!packed_ref_cache->lock)
+		die("internal error: packed refs not locked");
+	add_ref(get_packed_ref_dir(packed_ref_cache),
 		create_ref_entry(refname, sha1, REF_ISPACKED, 1));
 }
 
@@ -2031,6 +2049,56 @@ static int write_packed_entry_fn(struct ref_entry *entry, void *cb_data)
 	return 0;
 }
 
+int lock_packed_refs(struct lock_file *lock, int flags)
+{
+	int fd;
+	struct packed_ref_cache *packed_ref_cache;
+
+	/* Discard the old cache because it might be invalid: */
+	clear_packed_ref_cache(&ref_cache);
+	fd = hold_lock_file_for_update(lock, git_path("packed-refs"), flags);
+	if (fd < 0)
+		return -1;
+	/* Read the current packed-refs while holding the lock: */
+	packed_ref_cache = get_packed_ref_cache(&ref_cache);
+	packed_ref_cache->lock = lock;
+	packed_ref_cache->fd = fd;
+	return 0;
+}
+
+int commit_packed_refs(void)
+{
+	struct packed_ref_cache *packed_ref_cache =
+		get_packed_ref_cache(&ref_cache);
+	int fd = packed_ref_cache->fd;
+	int error = 0;
+
+	if (!packed_ref_cache->lock)
+		die("internal error: packed-refs not locked");
+	write_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
+
+	do_for_each_entry_in_dir(get_packed_ref_dir(packed_ref_cache),
+				 0, write_packed_entry_fn, &fd);
+	if (commit_lock_file(packed_ref_cache->lock))
+		error = -1;
+	packed_ref_cache->lock = NULL;
+	packed_ref_cache->fd = -1;
+	return error;
+}
+
+void rollback_packed_refs(void)
+{
+	struct packed_ref_cache *packed_ref_cache =
+		get_packed_ref_cache(&ref_cache);
+
+	if (!packed_ref_cache->lock)
+		die("internal error: packed-refs not locked");
+	rollback_lock_file(packed_ref_cache->lock);
+	packed_ref_cache->lock = NULL;
+	packed_ref_cache->fd = -1;
+	clear_packed_ref_cache(&ref_cache);
+}
+
 struct ref_to_prune {
 	struct ref_to_prune *next;
 	unsigned char sha1[20];
@@ -2149,23 +2217,19 @@ static struct lock_file packlock;
 int pack_refs(unsigned int flags)
 {
 	struct pack_refs_cb_data cbdata;
-	int fd;
 
 	memset(&cbdata, 0, sizeof(cbdata));
 	cbdata.flags = flags;
 
-	fd = hold_lock_file_for_update(&packlock, git_path("packed-refs"),
-				       LOCK_DIE_ON_ERROR);
+	lock_packed_refs(&packlock, LOCK_DIE_ON_ERROR);
 	cbdata.packed_refs = get_packed_refs(&ref_cache);
 
 	do_for_each_entry_in_dir(get_loose_refs(&ref_cache), 0,
 				 pack_if_possible_fn, &cbdata);
 
-	write_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
-	do_for_each_entry_in_dir(cbdata.packed_refs, 0, write_packed_entry_fn, &fd);
-
-	if (commit_lock_file(&packlock) < 0)
+	if (commit_packed_refs())
 		die_errno("unable to overwrite old ref-pack file");
+
 	prune_refs(cbdata.ref_to_prune);
 	return 0;
 }
@@ -2222,7 +2286,6 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
 
 static int repack_without_ref(const char *refname)
 {
-	int fd;
 	struct ref_dir *packed;
 	struct string_list refs_to_delete = STRING_LIST_INIT_DUP;
 	struct string_list_item *ref_to_delete;
@@ -2230,12 +2293,10 @@ static int repack_without_ref(const char *refname)
 	if (!get_packed_ref(refname))
 		return 0; /* refname does not exist in packed refs */
 
-	fd = hold_lock_file_for_update(&packlock, git_path("packed-refs"), 0);
-	if (fd < 0) {
+	if (lock_packed_refs(&packlock, 0)) {
 		unable_to_lock_error(git_path("packed-refs"), errno);
 		return error("cannot delete '%s' from packed refs", refname);
 	}
-	clear_packed_ref_cache(&ref_cache);
 	packed = get_packed_refs(&ref_cache);
 	/* Remove refname from the cache. */
 	if (remove_entry(packed, refname) == -1) {
@@ -2243,7 +2304,7 @@ static int repack_without_ref(const char *refname)
 		 * The packed entry disappeared while we were
 		 * acquiring the lock.
 		 */
-		rollback_lock_file(&packlock);
+		rollback_packed_refs();
 		return 0;
 	}
 	do_for_each_entry_in_dir(packed, 0, curate_packed_ref_fn, &refs_to_delete);
@@ -2251,9 +2312,7 @@ static int repack_without_ref(const char *refname)
 		if (remove_entry(packed, ref_to_delete->string) == -1)
 			die("internal error");
 	}
-	write_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));
-	do_for_each_entry_in_dir(packed, 0, write_packed_entry_fn, &fd);
-	return commit_lock_file(&packlock);
+	return commit_packed_refs();
 }
 
 int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
diff --git a/refs.h b/refs.h
index 246bf60..b275124 100644
--- a/refs.h
+++ b/refs.h
@@ -77,12 +77,35 @@ extern int for_each_rawref(each_ref_fn, void *);
 extern void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname);
 
 /*
- * Add a reference to the in-memory packed reference cache.  To actually
- * write the reference to the packed-refs file, call pack_refs().
+ * Lock the packed-refs file for writing using the specified lock_file
+ * instance.  Flags is passed to hold_lock_file_for_update().  Return
+ * 0 on success.
+ */
+extern int lock_packed_refs(struct lock_file *lock, int flags);
+
+/*
+ * Add a reference to the in-memory packed reference cache.  This may
+ * only be called while the packed-refs file is locked (see
+ * lock_packed_refs()).  To actually write the packed-refs file, call
+ * commit_packed_refs().
  */
 extern void add_packed_ref(const char *refname, const unsigned char *sha1);
 
 /*
+ * Write the current version of the packed refs cache from memory to
+ * disk.  The packed-refs file must already be locked for writing (see
+ * lock_packed_refs()).  Return zero on success.
+ */
+extern int commit_packed_refs(void);
+
+/*
+ * Rollback the lockfile for the packed-refs file, and discard the
+ * in-memory packed reference cache.  (The packed-refs file will be
+ * read anew if it is needed again after this function is called.)
+ */
+extern void rollback_packed_refs(void);
+
+/*
  * Flags for controlling behaviour of pack_refs()
  * PACK_REFS_PRUNE: Prune loose refs after packing
  * PACK_REFS_ALL:   Pack _all_ refs, not just tags and already packed refs

-- 
1.8.3
Jeff King· Jun 12, 2013, 12:01 UTC · re: Michael Haggerty · lore

Re: [PATCH 04/12] refs: implement simple transactions for the packed-refs file

On Tue, Jun 11, 2013 at 11:48:24PM +0200, Michael Haggerty wrote:
Show 9 quoted lines
> The API docs are not clear about whether it is kosher to read
> lock_file::fd directly.  It is only done in one file outside of
> lockfile.c.  So this patch stores the fd of the lockfile separately in
> struct packed_ref_cache, even though the same struct also has a
> pointer to the struct lock_file.
> 
> So please let me know if it is OK to read lock_file::fd directly.  If
> so, then I will drop the fd member of struct packed_ref_cache, as well
> as the local variable "fd" in lock_packed_refs().

I think it's fine; the fact that you have such an fd is a public part of the interface, so you are only relying on the struct member being there. And since the lock_file must hold the fd itself somewhere, I don't think that's unreasonable.

I'm not sure how you got your "in one file" list, but it appears to happen in credential-store.c, bundle.c, fast-import.c, and read-cache.c.

-Peff
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 05/12] refs: manage lifetime of packed refs cache via reference counting

In struct packed_ref_cache, keep a count of the number of users of the data structure. Only free the packed ref cache when the reference count goes to zero rather than when the packed ref cache is cleared. This mechanism will be used to prevent the cache data structure from being freed while it is being iterated over.

So far, only the reference in struct ref_cache::packed is counted; other users will be adjusted in separate commits.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)
Show changes to refs.c +36 −3
diff --git a/refs.c b/refs.c
index 114bd5b..b0d77a7 100644
--- a/refs.c
+++ b/refs.c
@@ -806,6 +806,14 @@ struct packed_ref_cache {
 	struct ref_entry *root;
 
 	/*
+	 * Count of references to the data structure in this instance,
+	 * including the pointer from ref_cache::packed if any.  The
+	 * data will not be freed as long as the reference count is
+	 * nonzero.
+	 */
+	unsigned int referrers;
+
+	/*
 	 * Iff the packed-refs file associated with this instance is
 	 * currently locked for writing, this points at the associated
 	 * lock (which is owned by somebody else).
@@ -832,14 +840,38 @@ static struct ref_cache {
 	char name[1];
 } ref_cache, *submodule_ref_caches;
 
+/*
+ * Increment the reference count of *packed_refs.
+ */
+static void acquire_packed_ref_cache(struct packed_ref_cache *packed_refs)
+{
+	packed_refs->referrers++;
+}
+
+/*
+ * Decrease the reference count of *packed_refs.  If it goes to zero,
+ * free *packed_refs and return true; otherwise return false.
+ */
+static int release_packed_ref_cache(struct packed_ref_cache *packed_refs)
+{
+	if (!--packed_refs->referrers) {
+		free_ref_entry(packed_refs->root);
+		free(packed_refs);
+		return 1;
+	} else {
+		return 0;
+	}
+}
+
 static void clear_packed_ref_cache(struct ref_cache *refs)
 {
 	if (refs->packed) {
-		if (refs->packed->lock)
+		struct packed_ref_cache *packed_refs = refs->packed;
+
+		if (packed_refs->lock)
 			die("internal error: packed-ref cache cleared while locked");
-		free_ref_entry(refs->packed->root);
-		free(refs->packed);
 		refs->packed = NULL;
+		release_packed_ref_cache(packed_refs);
 	}
 }
 
@@ -1020,6 +1052,7 @@ static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)
 		FILE *f;
 
 		refs->packed = xcalloc(1, sizeof(*refs->packed));
+		acquire_packed_ref_cache(refs->packed);
 		refs->packed->root = create_dir_entry(refs, "", 0, 0);
 		refs->packed->fd = -1;
 		if (*refs->name)

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 06/12] do_for_each_entry(): increment the packed refs cache refcount

This function calls a user-supplied callback function which could do something that causes the packed refs cache to be invalidated. So acquire a reference count on the data structure to prevent our copy from being freed while we are iterating over it.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Show changes to refs.c +4 −1
diff --git a/refs.c b/refs.c
index b0d77a7..d8e8ce2 100644
--- a/refs.c
+++ b/refs.c
@@ -1587,10 +1587,12 @@ void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)
 static int do_for_each_entry(struct ref_cache *refs, const char *base,
 			     each_ref_entry_fn fn, void *cb_data)
 {
-	struct ref_dir *packed_dir = get_packed_refs(refs);
+	struct packed_ref_cache *packed_ref_cache = get_packed_ref_cache(refs);
+	struct ref_dir *packed_dir = get_packed_ref_dir(packed_ref_cache);
 	struct ref_dir *loose_dir = get_loose_refs(refs);
 	int retval = 0;
 
+	acquire_packed_ref_cache(packed_ref_cache);
 	if (base && *base) {
 		packed_dir = find_containing_dir(packed_dir, base, 0);
 		loose_dir = find_containing_dir(loose_dir, base, 0);
@@ -1611,6 +1613,7 @@ static int do_for_each_entry(struct ref_cache *refs, const char *base,
 				loose_dir, 0, fn, cb_data);
 	}
 
+	release_packed_ref_cache(packed_ref_cache);
 	return retval;
 }
 

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 07/12] packed_ref_cache: increment refcount when locked

Increment the packed_ref_cache reference count while it is locked to prevent its being freed.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Show changes to refs.c +7 −1
diff --git a/refs.c b/refs.c
index d8e8ce2..92c8e97 100644
--- a/refs.c
+++ b/refs.c
@@ -816,7 +816,9 @@ struct packed_ref_cache {
 	/*
 	 * Iff the packed-refs file associated with this instance is
 	 * currently locked for writing, this points at the associated
-	 * lock (which is owned by somebody else).
+	 * lock (which is owned by somebody else).  The referrer count
+	 * is also incremented when the file is locked and decremented
+	 * when it is unlocked.
 	 */
 	struct lock_file *lock;
 
@@ -2099,6 +2101,8 @@ int lock_packed_refs(struct lock_file *lock, int flags)
 	packed_ref_cache = get_packed_ref_cache(&ref_cache);
 	packed_ref_cache->lock = lock;
 	packed_ref_cache->fd = fd;
+	/* Increment the reference count to prevent it from being freed: */
+	acquire_packed_ref_cache(packed_ref_cache);
 	return 0;
 }
 
@@ -2119,6 +2123,7 @@ int commit_packed_refs(void)
 		error = -1;
 	packed_ref_cache->lock = NULL;
 	packed_ref_cache->fd = -1;
+	release_packed_ref_cache(packed_ref_cache);
 	return error;
 }
 
@@ -2132,6 +2137,7 @@ void rollback_packed_refs(void)
 	rollback_lock_file(packed_ref_cache->lock);
 	packed_ref_cache->lock = NULL;
 	packed_ref_cache->fd = -1;
+	release_packed_ref_cache(packed_ref_cache);
 	clear_packed_ref_cache(&ref_cache);
 }
 

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 08/12] Extract a struct stat_data from cache_entry

Add public functions fill_stat_data() and match_stat_data() to work with it. This infrastructure will later be used to check the validity of other types of file.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
I'm not too familiar with this part of the code, so please make sure
that I've put the dividing line at the right place.
 builtin/ls-files.c |  12 +++--
 cache.h            |  33 +++++++++---
 read-cache.c       | 151 +++++++++++++++++++++++++++++------------------------
 3 files changed, 116 insertions(+), 80 deletions(-)
Show changes to 3 files +116 −80

builtin/ls-files.c, cache.h, read-cache.c

diff --git a/builtin/ls-files.c b/builtin/ls-files.c
index 2202072..6a0730f 100644
--- a/builtin/ls-files.c
+++ b/builtin/ls-files.c
@@ -165,11 +165,13 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)
 	}
 	write_name(ce->name, ce_namelen(ce));
 	if (debug_mode) {
-		printf("  ctime: %d:%d\n", ce->ce_ctime.sec, ce->ce_ctime.nsec);
-		printf("  mtime: %d:%d\n", ce->ce_mtime.sec, ce->ce_mtime.nsec);
-		printf("  dev: %d\tino: %d\n", ce->ce_dev, ce->ce_ino);
-		printf("  uid: %d\tgid: %d\n", ce->ce_uid, ce->ce_gid);
-		printf("  size: %d\tflags: %x\n", ce->ce_size, ce->ce_flags);
+		struct stat_data *sd = &ce->ce_stat_data;
+
+		printf("  ctime: %d:%d\n", sd->sd_ctime.sec, sd->sd_ctime.nsec);
+		printf("  mtime: %d:%d\n", sd->sd_mtime.sec, sd->sd_mtime.nsec);
+		printf("  dev: %d\tino: %d\n", sd->sd_dev, sd->sd_ino);
+		printf("  uid: %d\tgid: %d\n", sd->sd_uid, sd->sd_gid);
+		printf("  size: %d\tflags: %x\n", sd->sd_size, ce->ce_flags);
 	}
 }
 
diff --git a/cache.h b/cache.h
index df532f8..207f849 100644
--- a/cache.h
+++ b/cache.h
@@ -119,15 +119,19 @@ struct cache_time {
 	unsigned int nsec;
 };
 
+struct stat_data {
+	struct cache_time sd_ctime;
+	struct cache_time sd_mtime;
+	unsigned int sd_dev;
+	unsigned int sd_ino;
+	unsigned int sd_uid;
+	unsigned int sd_gid;
+	unsigned int sd_size;
+};
+
 struct cache_entry {
-	struct cache_time ce_ctime;
-	struct cache_time ce_mtime;
-	unsigned int ce_dev;
-	unsigned int ce_ino;
+	struct stat_data ce_stat_data;
 	unsigned int ce_mode;
-	unsigned int ce_uid;
-	unsigned int ce_gid;
-	unsigned int ce_size;
 	unsigned int ce_flags;
 	unsigned int ce_namelen;
 	unsigned char sha1[20];
@@ -509,6 +513,21 @@ extern int limit_pathspec_to_literal(void);
 #define HASH_FORMAT_CHECK 2
 extern int index_fd(unsigned char *sha1, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);
 extern int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags);
+
+/*
+ * Record to sd the data from st that we use to check whether a file
+ * might have changed.
+ */
+extern void fill_stat_data(struct stat_data *sd, struct stat *st);
+
+/*
+ * Return 0 if st is consistent with a file not having been changed
+ * since sd was filled.  If there are differences, return a
+ * combination of MTIME_CHANGED, CTIME_CHANGED, OWNER_CHANGED,
+ * INODE_CHANGED, and DATA_CHANGED.
+ */
+extern int match_stat_data(struct stat_data *sd, struct stat *st);
+
 extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
 
 #define REFRESH_REALLY		0x0001	/* ignore_valid */
diff --git a/read-cache.c b/read-cache.c
index 04ed561..4c4328e 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -67,6 +67,61 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n
 	add_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
 }
 
+void fill_stat_data(struct stat_data *sd, struct stat *st)
+{
+	sd->sd_ctime.sec = (unsigned int)st->st_ctime;
+	sd->sd_mtime.sec = (unsigned int)st->st_mtime;
+	sd->sd_ctime.nsec = ST_CTIME_NSEC(*st);
+	sd->sd_mtime.nsec = ST_MTIME_NSEC(*st);
+	sd->sd_dev = st->st_dev;
+	sd->sd_ino = st->st_ino;
+	sd->sd_uid = st->st_uid;
+	sd->sd_gid = st->st_gid;
+	sd->sd_size = st->st_size;
+}
+
+int match_stat_data(struct stat_data *sd, struct stat *st)
+{
+	int changed = 0;
+
+	if (sd->sd_mtime.sec != (unsigned int)st->st_mtime)
+		changed |= MTIME_CHANGED;
+	if (trust_ctime && check_stat &&
+	    sd->sd_ctime.sec != (unsigned int)st->st_ctime)
+		changed |= CTIME_CHANGED;
+
+#ifdef USE_NSEC
+	if (check_stat && sd->sd_mtime.nsec != ST_MTIME_NSEC(*st))
+		changed |= MTIME_CHANGED;
+	if (trust_ctime && check_stat &&
+	    sd->sd_ctime.nsec != ST_CTIME_NSEC(*st))
+		changed |= CTIME_CHANGED;
+#endif
+
+	if (check_stat) {
+		if (sd->sd_uid != (unsigned int) st->st_uid ||
+			sd->sd_gid != (unsigned int) st->st_gid)
+			changed |= OWNER_CHANGED;
+		if (sd->sd_ino != (unsigned int) st->st_ino)
+			changed |= INODE_CHANGED;
+	}
+
+#ifdef USE_STDEV
+	/*
+	 * st_dev breaks on network filesystems where different
+	 * clients will have different views of what "device"
+	 * the filesystem is on
+	 */
+	if (check_stat && sd->sd_dev != (unsigned int) st->st_dev)
+			changed |= INODE_CHANGED;
+#endif
+
+	if (sd->sd_size != (unsigned int) st->st_size)
+		changed |= DATA_CHANGED;
+
+	return changed;
+}
+
 /*
  * This only updates the "non-critical" parts of the directory
  * cache, ie the parts that aren't tracked by GIT, and only used
@@ -74,15 +129,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n
  */
 void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)
 {
-	ce->ce_ctime.sec = (unsigned int)st->st_ctime;
-	ce->ce_mtime.sec = (unsigned int)st->st_mtime;
-	ce->ce_ctime.nsec = ST_CTIME_NSEC(*st);
-	ce->ce_mtime.nsec = ST_MTIME_NSEC(*st);
-	ce->ce_dev = st->st_dev;
-	ce->ce_ino = st->st_ino;
-	ce->ce_uid = st->st_uid;
-	ce->ce_gid = st->st_gid;
-	ce->ce_size = st->st_size;
+	fill_stat_data(&ce->ce_stat_data, st);
 
 	if (assume_unchanged)
 		ce->ce_flags |= CE_VALID;
@@ -195,43 +242,11 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)
 	default:
 		die("internal error: ce_mode is %o", ce->ce_mode);
 	}
-	if (ce->ce_mtime.sec != (unsigned int)st->st_mtime)
-		changed |= MTIME_CHANGED;
-	if (trust_ctime && check_stat &&
-	    ce->ce_ctime.sec != (unsigned int)st->st_ctime)
-		changed |= CTIME_CHANGED;
 
-#ifdef USE_NSEC
-	if (check_stat && ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))
-		changed |= MTIME_CHANGED;
-	if (trust_ctime && check_stat &&
-	    ce->ce_ctime.nsec != ST_CTIME_NSEC(*st))
-		changed |= CTIME_CHANGED;
-#endif
-
-	if (check_stat) {
-		if (ce->ce_uid != (unsigned int) st->st_uid ||
-			ce->ce_gid != (unsigned int) st->st_gid)
-			changed |= OWNER_CHANGED;
-		if (ce->ce_ino != (unsigned int) st->st_ino)
-			changed |= INODE_CHANGED;
-	}
-
-#ifdef USE_STDEV
-	/*
-	 * st_dev breaks on network filesystems where different
-	 * clients will have different views of what "device"
-	 * the filesystem is on
-	 */
-	if (check_stat && ce->ce_dev != (unsigned int) st->st_dev)
-			changed |= INODE_CHANGED;
-#endif
-
-	if (ce->ce_size != (unsigned int) st->st_size)
-		changed |= DATA_CHANGED;
+	changed |= match_stat_data(&ce->ce_stat_data, st);
 
 	/* Racily smudged entry? */
-	if (!ce->ce_size) {
+	if (!ce->ce_stat_data.sd_size) {
 		if (!is_empty_blob_sha1(ce->sha1))
 			changed |= DATA_CHANGED;
 	}
@@ -245,11 +260,11 @@ static int is_racy_timestamp(const struct index_state *istate, struct cache_entr
 		istate->timestamp.sec &&
 #ifdef USE_NSEC
 		 /* nanosecond timestamped files can also be racy! */
-		(istate->timestamp.sec < ce->ce_mtime.sec ||
-		 (istate->timestamp.sec == ce->ce_mtime.sec &&
-		  istate->timestamp.nsec <= ce->ce_mtime.nsec))
+		(istate->timestamp.sec < ce->ce_stat_data.sd_mtime.sec ||
+		 (istate->timestamp.sec == ce->ce_stat_data.sd_mtime.sec &&
+		  istate->timestamp.nsec <= ce->ce_stat_data.sd_mtime.nsec))
 #else
-		istate->timestamp.sec <= ce->ce_mtime.sec
+		istate->timestamp.sec <= ce->ce_stat_data.sd_mtime.sec
 #endif
 		 );
 }
@@ -340,7 +355,7 @@ int ie_modified(const struct index_state *istate,
 	 * then we know it is.
 	 */
 	if ((changed & DATA_CHANGED) &&
-	    (S_ISGITLINK(ce->ce_mode) || ce->ce_size != 0))
+	    (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0))
 		return changed;
 
 	changed_fs = ce_modified_check_fs(ce, st);
@@ -1322,16 +1337,16 @@ static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *on
 {
 	struct cache_entry *ce = xmalloc(cache_entry_size(len));
 
-	ce->ce_ctime.sec = ntoh_l(ondisk->ctime.sec);
-	ce->ce_mtime.sec = ntoh_l(ondisk->mtime.sec);
-	ce->ce_ctime.nsec = ntoh_l(ondisk->ctime.nsec);
-	ce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);
-	ce->ce_dev   = ntoh_l(ondisk->dev);
-	ce->ce_ino   = ntoh_l(ondisk->ino);
+	ce->ce_stat_data.sd_ctime.sec = ntoh_l(ondisk->ctime.sec);
+	ce->ce_stat_data.sd_mtime.sec = ntoh_l(ondisk->mtime.sec);
+	ce->ce_stat_data.sd_ctime.nsec = ntoh_l(ondisk->ctime.nsec);
+	ce->ce_stat_data.sd_mtime.nsec = ntoh_l(ondisk->mtime.nsec);
+	ce->ce_stat_data.sd_dev   = ntoh_l(ondisk->dev);
+	ce->ce_stat_data.sd_ino   = ntoh_l(ondisk->ino);
 	ce->ce_mode  = ntoh_l(ondisk->mode);
-	ce->ce_uid   = ntoh_l(ondisk->uid);
-	ce->ce_gid   = ntoh_l(ondisk->gid);
-	ce->ce_size  = ntoh_l(ondisk->size);
+	ce->ce_stat_data.sd_uid   = ntoh_l(ondisk->uid);
+	ce->ce_stat_data.sd_gid   = ntoh_l(ondisk->gid);
+	ce->ce_stat_data.sd_size  = ntoh_l(ondisk->size);
 	ce->ce_flags = flags & ~CE_NAMEMASK;
 	ce->ce_namelen = len;
 	hashcpy(ce->sha1, ondisk->sha1);
@@ -1608,7 +1623,7 @@ static void ce_smudge_racily_clean_entry(struct cache_entry *ce)
 	 * The only thing we care about in this function is to smudge the
 	 * falsely clean entry due to touch-update-touch race, so we leave
 	 * everything else as they are.  We are called for entries whose
-	 * ce_mtime match the index file mtime.
+	 * ce_stat_data.sd_mtime match the index file mtime.
 	 *
 	 * Note that this actually does not do much for gitlinks, for
 	 * which ce_match_stat_basic() always goes to the actual
@@ -1647,7 +1662,7 @@ static void ce_smudge_racily_clean_entry(struct cache_entry *ce)
 		 * file, and never calls us, so the cached size information
 		 * for "frotz" stays 6 which does not match the filesystem.
 		 */
-		ce->ce_size = 0;
+		ce->ce_stat_data.sd_size = 0;
 	}
 }
 
@@ -1657,16 +1672,16 @@ static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,
 {
 	short flags;
 
-	ondisk->ctime.sec = htonl(ce->ce_ctime.sec);
-	ondisk->mtime.sec = htonl(ce->ce_mtime.sec);
-	ondisk->ctime.nsec = htonl(ce->ce_ctime.nsec);
-	ondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);
-	ondisk->dev  = htonl(ce->ce_dev);
-	ondisk->ino  = htonl(ce->ce_ino);
+	ondisk->ctime.sec = htonl(ce->ce_stat_data.sd_ctime.sec);
+	ondisk->mtime.sec = htonl(ce->ce_stat_data.sd_mtime.sec);
+	ondisk->ctime.nsec = htonl(ce->ce_stat_data.sd_ctime.nsec);
+	ondisk->mtime.nsec = htonl(ce->ce_stat_data.sd_mtime.nsec);
+	ondisk->dev  = htonl(ce->ce_stat_data.sd_dev);
+	ondisk->ino  = htonl(ce->ce_stat_data.sd_ino);
 	ondisk->mode = htonl(ce->ce_mode);
-	ondisk->uid  = htonl(ce->ce_uid);
-	ondisk->gid  = htonl(ce->ce_gid);
-	ondisk->size = htonl(ce->ce_size);
+	ondisk->uid  = htonl(ce->ce_stat_data.sd_uid);
+	ondisk->gid  = htonl(ce->ce_stat_data.sd_gid);
+	ondisk->size = htonl(ce->ce_stat_data.sd_size);
 	hashcpy(ondisk->sha1, ce->sha1);
 
 	flags = ce->ce_flags;

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 09/12] add a stat_validity struct

It can sometimes be useful to know whether a path in the filesystem has been updated without going to the work of opening and re-reading its content. We trust the stat() information on disk already to handle index updates, and we can use the same trick here.

This patch introduces a "stat_validity" struct which encapsulates the concept of checking the stat-freshness of a file. It is implemented on top of "struct stat_data" to reuse the logic about which stat entries to trust for a particular platform, but hides the complexity behind two simple functions: check and update.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
This is *very* similar to a patch by Jeff King <peff@peff.net> [1],
except that it is based on the struct stat_data that I extracted from
cache_entry rather than using cache_entries directly.  I would have
left Peff the author except that I don't want to risk putting him on
the hook for any mistakes that I might have made.  But if it is
appropriate, don't hesitate to make him author again.
[1] http://article.gmane.org/gmane.comp.version-control.git/223526
 cache.h      | 27 +++++++++++++++++++++++++++
 read-cache.c | 30 ++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)
Show changes to 2 files +57 −0

cache.h, read-cache.c

diff --git a/cache.h b/cache.h
index 207f849..15f5110 100644
--- a/cache.h
+++ b/cache.h
@@ -1358,4 +1358,31 @@ int checkout_fast_forward(const unsigned char *from,
 
 int sane_execvp(const char *file, char *const argv[]);
 
+/*
+ * A struct to encapsulate the concept of whether a file has changed
+ * since we last checked it. This uses criteria similar to those used
+ * for the index.
+ */
+struct stat_validity {
+	struct stat_data *sd;
+};
+
+void stat_validity_clear(struct stat_validity *sv);
+
+/*
+ * Returns 1 if the path is a regular file (or a symlink to a regular
+ * file) and matches the saved stat_validity, 0 otherwise.  A missing
+ * or inaccessible file is considered a match if the struct was just
+ * initialized, or if the previous update found an inaccessible file.
+ */
+int stat_validity_check(struct stat_validity *sv, const char *path);
+
+/*
+ * Update the stat_validity from a file opened at descriptor fd. If
+ * the file is missing, inaccessible, or not a regular file, then
+ * future calls to stat_validity_check will match iff one of those
+ * conditions continues to be true.
+ */
+void stat_validity_update(struct stat_validity *sv, int fd);
+
 #endif /* CACHE_H */
diff --git a/read-cache.c b/read-cache.c
index 4c4328e..73e85a4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1948,3 +1948,33 @@ void *read_blob_data_from_index(struct index_state *istate, const char *path, un
 		*size = sz;
 	return data;
 }
+
+void stat_validity_clear(struct stat_validity *sv)
+{
+	free(sv->sd);
+	sv->sd = NULL;
+}
+
+int stat_validity_check(struct stat_validity *sv, const char *path)
+{
+	struct stat st;
+
+	if (stat(path, &st) < 0)
+		return sv->sd == NULL;
+	if (!sv->sd)
+		return 0;
+	return S_ISREG(st.st_mode) && !match_stat_data(sv->sd, &st);
+}
+
+void stat_validity_update(struct stat_validity *sv, int fd)
+{
+	struct stat st;
+
+	if (fstat(fd, &st) < 0 || !S_ISREG(st.st_mode))
+		stat_validity_clear(sv);
+	else {
+		if (!sv->sd)
+			sv->sd = xcalloc(1, sizeof(struct stat_data));
+		fill_stat_data(sv->sd, &st);
+	}
+}

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 10/12] get_packed_ref_cache: reload packed-refs file when it changes

From: Jeff King <peff@peff.net>

Once we read the packed-refs file into memory, we cache it to save work on future ref lookups. However, our cache may be out of date with respect to what is on disk if another process is simultaneously packing the refs. Normally it is acceptable for us to be a little out of date, since there is no guarantee whether we read the file before or after the simultaneous update. However, there is an important special case: our packed-refs file must be up to date with respect to any loose refs we read. Otherwise, we risk the following race condition:

  0. There exists a loose ref refs/heads/master.
  1. Process A starts and looks up the ref "master". It
     first checks $GIT_DIR/master, which does not exist. It
     then loads (and caches) the packed-refs file to see if
     "master" exists in it, which it does not.
  2. Meanwhile, process B runs "pack-refs --all --prune". It
     creates a new packed-refs file which contains
     refs/heads/master, and removes the loose copy at
     $GIT_DIR/refs/heads/master.
  3. Process A continues its lookup, and eventually tries
     $GIT_DIR/refs/heads/master.  It sees that the loose ref
     is missing, and falls back to the packed-refs file. But
     it examines its cached version, which does not have
     refs/heads/master. After trying a few other prefixes,
     it reports master as a non-existent ref.

There are many variants (e.g., step 1 may involve process A looking up another ref entirely, so even a fully qualified refname can fail). One of the most interesting ones is if "refs/heads/master" is already packed. In that case process A will not see it as missing, but rather will report whatever value happened to be in the packed-refs file before process B repacked (which might be an arbitrarily old value).

We can fix this by making sure we reload the packed-refs file from disk after looking at any loose refs. That's unacceptably slow, so we can check its stat()-validity as a proxy, and read it only when it appears to have changed.

Reading the packed-refs file after performing any loose-ref system calls is sufficient because we know the ordering of the pack-refs process: it always makes sure the newly written packed-refs file is installed into place before pruning any loose refs. As long as those operations by B appear in their executed order to process A, by the time A sees the missing loose ref, the new packed-refs file must be in place.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
This is Peff's work, rebased and with some smallish changes to fit it
in with the packed_ref_cache data structure.
 refs.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
Show changes to refs.c +16 −5
diff --git a/refs.c b/refs.c
index 92c8e97..64f72ab 100644
--- a/refs.c
+++ b/refs.c
@@ -824,6 +824,9 @@ struct packed_ref_cache {
 
 	/* If locked, the file descriptor of the lock file. */
 	int fd;
+
+	/* The metadata from when this packed-refs cache was read */
+	struct stat_validity validity;
 };
 
 /*
@@ -858,6 +861,7 @@ static int release_packed_ref_cache(struct packed_ref_cache *packed_refs)
 {
 	if (!--packed_refs->referrers) {
 		free_ref_entry(packed_refs->root);
+		stat_validity_clear(&packed_refs->validity);
 		free(packed_refs);
 		return 1;
 	} else {
@@ -1049,20 +1053,27 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)
  */
 static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)
 {
+	const char *packed_refs_file;
+
+	if (*refs->name)
+		packed_refs_file = git_path_submodule(refs->name, "packed-refs");
+	else
+		packed_refs_file = git_path("packed-refs");
+
+	if (refs->packed &&
+	    !stat_validity_check(&refs->packed->validity, packed_refs_file))
+		clear_packed_ref_cache(refs);
+
 	if (!refs->packed) {
-		const char *packed_refs_file;
 		FILE *f;
 
 		refs->packed = xcalloc(1, sizeof(*refs->packed));
 		acquire_packed_ref_cache(refs->packed);
 		refs->packed->root = create_dir_entry(refs, "", 0, 0);
 		refs->packed->fd = -1;
-		if (*refs->name)
-			packed_refs_file = git_path_submodule(refs->name, "packed-refs");
-		else
-			packed_refs_file = git_path("packed-refs");
 		f = fopen(packed_refs_file, "r");
 		if (f) {
+			stat_validity_update(&refs->packed->validity, fileno(f));
 			read_packed_refs(f, get_ref_dir(refs->packed->root));
 			fclose(f);
 		}

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 11/12] for_each_ref: load all loose refs before packed refs

From: Jeff King <peff@peff.net>

If we are iterating through the refs using for_each_ref (or any of its sister functions), we can get into a race condition with a simultaneous "pack-refs --prune" that looks like this:

  0. We have a large number of loose refs, and a few packed
     refs. refs/heads/z/foo is loose, with no matching entry
     in the packed-refs file.
  1. Process A starts iterating through the refs. It loads
     the packed-refs file from disk, then starts lazily
     traversing through the loose ref directories.
  2. Process B, running "pack-refs --prune", writes out the
     new packed-refs file. It then deletes the newly packed
     refs, including refs/heads/z/foo.
  3. Meanwhile, process A has finally gotten to
     refs/heads/z (it traverses alphabetically). It
     descends, but finds nothing there.  It checks its
     cached view of the packed-refs file, but it does not
     mention anything in "refs/heads/z/" at all (it predates
     the new file written by B in step 2).

The traversal completes successfully without mentioning refs/heads/z/foo at all (the name, of course, isn't important; but the more refs you have and the farther down the alphabetical list a ref is, the more likely it is to hit the race). If refs/heads/z/foo did exist in the packed refs file at state 0, we would see an entry for it, but it would show whatever sha1 the ref had the last time it was packed (which could be an arbitrarily long time ago).

This can be especially dangerous when process A is "git prune", as it means our set of reachable tips will be incomplete, and we may erroneously prune objects reachable from that tip (the same thing can happen if "repack -ad" is used, as it simply drops unreachable objects that are packed).

This patch solves it by loading all of the loose refs for our traversal into our in-memory cache, and then refreshing the packed-refs cache. Because a pack-refs writer will always put the new packed-refs file into place before starting the prune, we know that any loose refs we fail to see will either truly be missing, or will have already been put in the packed-refs file by the time we refresh.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
Ditto.
 refs.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
Show changes to refs.c +35 −4
diff --git a/refs.c b/refs.c
index 64f72ab..aa4641b 100644
--- a/refs.c
+++ b/refs.c
@@ -746,6 +746,21 @@ static int do_for_each_entry_in_dirs(struct ref_dir *dir1,
 }
 
 /*
+ * Load all of the refs from the dir into our in-memory cache. The hard work
+ * of loading loose refs is done by get_ref_dir(), so we just need to recurse
+ * through all of the sub-directories. We do not even need to care about
+ * sorting, as traversal order does not matter to us.
+ */
+static void prime_ref_dir(struct ref_dir *dir)
+{
+	int i;
+	for (i = 0; i < dir->nr; i++) {
+		struct ref_entry *entry = dir->entries[i];
+		if (entry->flag & REF_DIR)
+			prime_ref_dir(get_ref_dir(entry));
+	}
+}
+/*
  * Return true iff refname1 and refname2 conflict with each other.
  * Two reference names conflict if one of them exactly matches the
  * leading components of the other; e.g., "foo/bar" conflicts with
@@ -1600,15 +1615,31 @@ void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)
 static int do_for_each_entry(struct ref_cache *refs, const char *base,
 			     each_ref_entry_fn fn, void *cb_data)
 {
-	struct packed_ref_cache *packed_ref_cache = get_packed_ref_cache(refs);
-	struct ref_dir *packed_dir = get_packed_ref_dir(packed_ref_cache);
-	struct ref_dir *loose_dir = get_loose_refs(refs);
+	struct packed_ref_cache *packed_ref_cache;
+	struct ref_dir *loose_dir;
+	struct ref_dir *packed_dir;
 	int retval = 0;
 
+	/*
+	 * We must make sure that all loose refs are read before accessing the
+	 * packed-refs file; this avoids a race condition in which loose refs
+	 * are migrated to the packed-refs file by a simultaneous process, but
+	 * our in-memory view is from before the migration. get_packed_ref_cache()
+	 * takes care of making sure our view is up to date with what is on
+	 * disk.
+	 */
+	loose_dir = get_loose_refs(refs);
+	if (base && *base) {
+		loose_dir = find_containing_dir(loose_dir, base, 0);
+	}
+	if (loose_dir)
+		prime_ref_dir(loose_dir);
+
+	packed_ref_cache = get_packed_ref_cache(refs);
 	acquire_packed_ref_cache(packed_ref_cache);
+	packed_dir = get_packed_ref_dir(packed_ref_cache);
 	if (base && *base) {
 		packed_dir = find_containing_dir(packed_dir, base, 0);
-		loose_dir = find_containing_dir(loose_dir, base, 0);
 	}
 
 	if (packed_dir && loose_dir) {

-- 
1.8.3
Michael Haggerty· Jun 11, 2013, 21:48 UTC · re: Michael Haggerty · lore

[PATCH 12/12] refs: do not invalidate the packed-refs cache unnecessarily

Now that we keep track of the packed-refs file metadata, we can detect when the packed-refs file has been modified since we last read it, and we do so automatically every time that get_packed_ref_cache() is called. So there is no need to invalidate the cache automatically when lock_packed_refs() is called; usually the old copy will still be valid.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
This patch is optional.  It makes the assumption that the metadata
stored in stat_validity are adequate to reliably detect when the
packed-refs file has changed.  Given that we are about to rewrite the
file, it is perhaps even more crucial not to make a mistake in this
codepath than in others.  So if the stat_validity check is not
considered safe enough, it might be prudent to omit this patch and
continue to reload the packed-refs data here unconditionally.
 refs.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Show changes to refs.c +6 −3
diff --git a/refs.c b/refs.c
index aa4641b..d9cbfc4 100644
--- a/refs.c
+++ b/refs.c
@@ -2134,12 +2134,15 @@ int lock_packed_refs(struct lock_file *lock, int flags)
 	int fd;
 	struct packed_ref_cache *packed_ref_cache;
 
-	/* Discard the old cache because it might be invalid: */
-	clear_packed_ref_cache(&ref_cache);
 	fd = hold_lock_file_for_update(lock, git_path("packed-refs"), flags);
 	if (fd < 0)
 		return -1;
-	/* Read the current packed-refs while holding the lock: */
+	/*
+	 * Get the current packed-refs while holding the lock.  If the
+	 * packed-refs file has been modified since we last read it,
+	 * this will automatically invalidate the cache and re-read
+	 * the packed-refs file.
+	 */
 	packed_ref_cache = get_packed_ref_cache(&ref_cache);
 	packed_ref_cache->lock = lock;
 	packed_ref_cache->fd = fd;

-- 
1.8.3
Jeff King· Jun 12, 2013, 12:39 UTC · re: Michael Haggerty · lore

Re: [PATCH 12/12] refs: do not invalidate the packed-refs cache unnecessarily

On Tue, Jun 11, 2013 at 11:48:32PM +0200, Michael Haggerty wrote:
Show 16 quoted lines
> Now that we keep track of the packed-refs file metadata, we can detect
> when the packed-refs file has been modified since we last read it, and
> we do so automatically every time that get_packed_ref_cache() is
> called.  So there is no need to invalidate the cache automatically
> when lock_packed_refs() is called; usually the old copy will still be
> valid.
> 
> Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
> ---
> This patch is optional.  It makes the assumption that the metadata
> stored in stat_validity are adequate to reliably detect when the
> packed-refs file has changed.  Given that we are about to rewrite the
> file, it is perhaps even more crucial not to make a mistake in this
> codepath than in others.  So if the stat_validity check is not
> considered safe enough, it might be prudent to omit this patch and
> continue to reload the packed-refs data here unconditionally.

I doubt that the risk is very high, but I'd also doubt that this provides any useful performance improvement, because it is mostly happening during a `git pack-refs` call. The exception is packing for a ref deletion, and it could potentially speed up a receive-pack that was deleting many refs.

I don't have a strong opinion either way.
-Peff
Jeff King· Jun 12, 2013, 12:52 UTC · re: Michael Haggerty · lore

Re: [PATCH 00/12] Fix some reference-related races

On Tue, Jun 11, 2013 at 11:48:20PM +0200, Michael Haggerty wrote:
> *This patch series must be built on top of mh/reflife.*

Applying on top of what Junio has in mh/reflife seems to create conflicts at the first patch. I didn't look into it, though, but just read the patches and looked at the packed-refs-transactions branch at git://github.com/mhagger/git.git.

Show 5 quoted lines
> The other problem was that the for_each_ref() functions will die if
> the ref cache that they are iterating over is freed out from under
> them.  This problem is solved by using reference counts to avoid
> freeing the old packed ref cache (even if it is no longer valid) until
> all users are done with it.

I was worried that the reference-counting would end up invasive and ugly, but it turned out quite nice.

Overall, I think the series goes in the right direction, and I didn't see any implementation flaws. The core of the race-handling is the same as in mine, but your "hyperactive repository" patch shows how badly mine can break under load.

I sent a few comments to specific patches, but I'd be fine with applying it as-is. Thanks for working on it.

-Peff
Ramsay Jones· Jun 15, 2013, 20:13 UTC · re: Michael Haggerty · lore

Re: [PATCH 00/12] Fix some reference-related races

Michael Haggerty wrote:
> *This patch series must be built on top of mh/reflife.*
> 
[...]
Show 5 quoted lines
> The other problem was that the for_each_ref() functions will die if
> the ref cache that they are iterating over is freed out from under
> them.  This problem is solved by using reference counts to avoid
> freeing the old packed ref cache (even if it is no longer valid) until
> all users are done with it.

Yes, I found exactly this happened to me on cygwin, earlier this week, with the previous version of this code. After seeing this mail, I had decided not to describe the failure on the old version, but wait and test this version instead.

This version is a great improvement, but it still has some failures on cygwin. So, it may be worth (briefly) describing the old failure anyway! Note that several tests failed, but I will only mention t3211-peel-ref.sh tests #7-8.

  $ pwd
  /home/ramsay/git/t/trash directory.t3211-peel-ref
  $
  $ ../../bin-wrappers/git show-ref -d
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
        5 [main] git 3540 _cygtls::handle_exceptions: Error while dumping state (p
  robably corrupted stack)
  Segmentation fault (core dumped)
  $
The stack-trace for the faulting code looks something like:
  cmd_show_ref()
    for_each_ref(show_ref, NULL)
      do_for_each_ref(&ref_cache, "", show_ref, 0, 0, NULL)
        do_for_each_entry(&ref_cache, "", do_one_ref, &data)
          do_for_each_entry_in_dirs(packed_dir, loose_dir, do_one_ref, &data)
            *dfeeid() recursive calls*
              do_one_ref(entry, &data)
                show_ref("refs/outside/foo", sha1, NULL) [2nd match]
                  peel_ref("refs/outside/foo", sha1)
                    peel_entry(entry, 0)
                      peel_object(name, sha1)
                        deref_tag_noverify(o)
                          parse_object(sha1 <eb0e854c2...>)
                            lookup_replace_object(sha1)
                              do_lookup_replace_object(sha1)
                                prepare_replace_object()
  [un-indent here!]
      for_each_replace_ref(register_replace_ref, NULL)
        do_for_each_ref(&ref_cache, "refs/replace", fn, 13, 0, NULL)
          do_for_each_entry(&ref_cache, "refs/replace", fn, &data)
            get_packed_refs(&ref_cache)
              clear_packed_ref_cache(&ref_cache) *free_ref_entries etc*
  ** return to show_ref() [2nd match] above **
  ** return to recursive dfeeid() call in original iteration
  ** dir1->entries has been free()-ed and reused => segmentation fault
  [dir1->entries == 0x64633263 => dc2c => part of sha1 for refs/outside/foo]

So, the nested "replace-reference-iteration" causes the ref_cache to be freed out from under the initial show-ref iteration, so this works:

  $ GIT_NO_REPLACE_OBJECTS=1 ../../bin-wrappers/git show-ref -d
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/foo^{}
  $

You may be wondering why clear_packed_ref_cache() is called? Well, that is because stat_validity_check() *incorrectly* indicates that the packed-refs file has changed. Why does it do that? Well, this is one more example of the problems caused by the cygwin schizophrenic stat() functions. :( [ARGHHHHHHHHH]

At this point, I tried running 'git show-ref' with core.checkstat set on the command line; but that didn't work! I had to fix show-ref and re-build git, and then, this works:

  $ ../../bin-wrappers/git -c core.checkstat=minimal -c core.trustctime=f
  alse show-ref -d
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/foo^{}
  $

Now, turning to the new code, t3211-peel-ref.sh test #7 now works, but test #8 still fails...

  $ ./t3211-peel-ref.sh -i -v
  ...
  ok 7 - refs are peeled outside of refs/tags (old packed)
  expecting success:
          git pack-refs --all &&
          cp .git/packed-refs fully-peeled &&
          git branch yadda &&
          git pack-refs --all &&
          git branch -d yadda &&
          test_cmp fully-peeled .git/packed-refs
  fatal: internal error: packed-ref cache cleared while locked
  not ok 8 - peeled refs survive deletion of packed ref
  #
  #               git pack-refs --all &&
  #               cp .git/packed-refs fully-peeled &&
  #               git branch yadda &&
  #               git pack-refs --all &&
  #               git branch -d yadda &&
  #               test_cmp fully-peeled .git/packed-refs
  #
  $ cd trash\ directory.t3211-peel-ref/
  $ ../../bin-wrappers/git pack-refs --all
  fatal: internal error: packed-ref cache cleared while locked
  $ ls
  actual  base.t  expect
  $ ls .git
  COMMIT_EDITMSG  branches/  description      index  logs/     packed-refs
  HEAD            config     hooks-disabled/  info/  objects/  refs/
  $ ls -l .git/packed-refs
  -rw-r--r-- 1 ramsay None 296 Jun 14 20:34 .git/packed-refs
  $ cat .git/packed-refs
  # pack-refs with: peeled
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
  ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
  $

Now, I have a test-stat program which prints the difference between the two stat implementations used in cygwin git, thus:

  $ ../../test-stat .git/packed-refs
  stat for '.git/packed-refs':
  *dev:   -1395862925, 0
  *ino:   166044, 0
  *mode:  100644 -rw-, 100600 -rw-
   nlink: 1, 1
  *uid:   1005, 0
  *gid:   513, 0
  *rdev:  -1395862925, 0
   size:  296, 296
   atime: 1371238550, 1371238550 Fri Jun 14 20:35:50 2013
   mtime: 1371238469, 1371238469 Fri Jun 14 20:34:29 2013
   ctime: 1371238469, 1371238469 Fri Jun 14 20:34:29 2013
  $ ../../bin-wrappers/git -c core.checkstat=minimal pack-refs --all
  fatal: internal error: packed-ref cache cleared while locked
  $

Hmmm, that should have worked! Wait, fix 'git pack-refs' to support setting config variables on the command line, rebuild and:

  $ ../../bin-wrappers/git -c core.checkstat=minimal pack-refs --all
  $ cat .git/packed-refs
  # pack-refs with: peeled fully-peeled
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
  ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
  d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
  eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
  ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
  $

I haven't checked the remaining test failures to see if they are caused by this code (I don't think so, but ...), but this failure is clearly a cygwin specific issue.

HTH

ATB, Ramsay Jones

Michael Haggerty· Jun 16, 2013, 05:50 UTC · re: Ramsay Jones · lore

Re: [PATCH 00/12] Fix some reference-related races

Thanks for all of the information.
On 06/15/2013 10:13 PM, Ramsay Jones wrote:
Show 96 quoted lines
> Michael Haggerty wrote:
>> *This patch series must be built on top of mh/reflife.*
>>
> 
> [...]
> 
>> The other problem was that the for_each_ref() functions will die if
>> the ref cache that they are iterating over is freed out from under
>> them.  This problem is solved by using reference counts to avoid
>> freeing the old packed ref cache (even if it is no longer valid) until
>> all users are done with it.
> 
> Yes, I found exactly this happened to me on cygwin, earlier this week,
> with the previous version of this code. After seeing this mail, I had
> decided not to describe the failure on the old version, but wait and
> test this version instead.
> 
> This version is a great improvement, but it still has some failures on
> cygwin. So, it may be worth (briefly) describing the old failure anyway!
> Note that several tests failed, but I will only mention t3211-peel-ref.sh
> tests #7-8.
> 
>   $ pwd
>   /home/ramsay/git/t/trash directory.t3211-peel-ref
>   $
> 
>   $ ../../bin-wrappers/git show-ref -d
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
>         5 [main] git 3540 _cygtls::handle_exceptions: Error while dumping state (p
>   robably corrupted stack)
>   Segmentation fault (core dumped)
>   $
> 
> The stack-trace for the faulting code looks something like:
> 
>   cmd_show_ref()
>     for_each_ref(show_ref, NULL)
>       do_for_each_ref(&ref_cache, "", show_ref, 0, 0, NULL)
>         do_for_each_entry(&ref_cache, "", do_one_ref, &data)
>           do_for_each_entry_in_dirs(packed_dir, loose_dir, do_one_ref, &data)
>             *dfeeid() recursive calls*
>               do_one_ref(entry, &data)
>                 show_ref("refs/outside/foo", sha1, NULL) [2nd match]
>                   peel_ref("refs/outside/foo", sha1)
>                     peel_entry(entry, 0)
>                       peel_object(name, sha1)
>                         deref_tag_noverify(o)
>                           parse_object(sha1 <eb0e854c2...>)
>                             lookup_replace_object(sha1)
>                               do_lookup_replace_object(sha1)
>                                 prepare_replace_object()
> 
>   [un-indent here!]
>       for_each_replace_ref(register_replace_ref, NULL)
>         do_for_each_ref(&ref_cache, "refs/replace", fn, 13, 0, NULL)
>           do_for_each_entry(&ref_cache, "refs/replace", fn, &data)
>             get_packed_refs(&ref_cache)
>               clear_packed_ref_cache(&ref_cache) *free_ref_entries etc*
>   ** return to show_ref() [2nd match] above **
>   ** return to recursive dfeeid() call in original iteration
>   ** dir1->entries has been free()-ed and reused => segmentation fault
>   [dir1->entries == 0x64633263 => dc2c => part of sha1 for refs/outside/foo]
> 
> So, the nested "replace-reference-iteration" causes the ref_cache to be
> freed out from under the initial show-ref iteration, so this works:
> 
>   $ GIT_NO_REPLACE_OBJECTS=1 ../../bin-wrappers/git show-ref -d
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/foo^{}
>   $
> 
> You may be wondering why clear_packed_ref_cache() is called? Well, that
> is because stat_validity_check() *incorrectly* indicates that the
> packed-refs file has changed. Why does it do that? Well, this is one
> more example of the problems caused by the cygwin schizophrenic stat()
> functions. :( [ARGHHHHHHHHH]
> 
> At this point, I tried running 'git show-ref' with core.checkstat set
> on the command line; but that didn't work! I had to fix show-ref and
> re-build git, and then, this works:
> 
>   $ ../../bin-wrappers/git -c core.checkstat=minimal -c core.trustctime=f
>   alse show-ref -d
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/outside/foo^{}
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/foo^{}
>   $

So if I understand correctly, all of the above is *without* the refcounting changes introduced in mh/ref-races? Is so, then it is not surprising, as this is exactly the sort of problem that the reference counting is meant to solve.

Show 30 quoted lines
> Now, turning to the new code, t3211-peel-ref.sh test #7 now works, but
> test #8 still fails...
> 
>   $ ./t3211-peel-ref.sh -i -v
> 
>   ...
> 
>   ok 7 - refs are peeled outside of refs/tags (old packed)
> 
>   expecting success:
>           git pack-refs --all &&
>           cp .git/packed-refs fully-peeled &&
>           git branch yadda &&
>           git pack-refs --all &&
>           git branch -d yadda &&
>           test_cmp fully-peeled .git/packed-refs
> 
>   fatal: internal error: packed-ref cache cleared while locked
>   not ok 8 - peeled refs survive deletion of packed ref
>   #
>   #               git pack-refs --all &&
>   #               cp .git/packed-refs fully-peeled &&
>   #               git branch yadda &&
>   #               git pack-refs --all &&
>   #               git branch -d yadda &&
>   #               test_cmp fully-peeled .git/packed-refs
>   #
>   $ cd trash\ directory.t3211-peel-ref/
>   $ ../../bin-wrappers/git pack-refs --all
>   fatal: internal error: packed-ref cache cleared while locked

These "internal error: packed-ref cache cleared while locked" failures result from an internal consistency check that clear_packed_ref_cache() is not called while the write lock is held on the packed-refs file. A call to c_p_r_c() could result from

* a programming error
* a determination based on the packed-refs file stat that the file needs
to be re-read

Judging from what you said about cygwin, I assume that the latter is happening. It should be impossible, because the current process is holding packed-refs.lock, and therefore other git processes should refuse to change the packed-refs file.

But if the stat information is not reliable, then the current process would *think* that the packed-refs file has been changed even though it hasn't, then it would call c_p_r_c() even though it holds the lock on packed-refs, and the internal consistency check would fail.

So apparently in these cases cygwin is reporting that the packed-refs stat information has changed (in the sense defined by the new stat_validity_check() function, which does essentially the same checks as the old ce_match_stat_basic() function).

Show 32 quoted lines
>   $ ls
>   actual  base.t  expect
>   $ ls .git
>   COMMIT_EDITMSG  branches/  description      index  logs/     packed-refs
>   HEAD            config     hooks-disabled/  info/  objects/  refs/
>   $ ls -l .git/packed-refs
>   -rw-r--r-- 1 ramsay None 296 Jun 14 20:34 .git/packed-refs
>   $ cat .git/packed-refs
>   # pack-refs with: peeled
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
>   ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
>   $
> 
> Now, I have a test-stat program which prints the difference between
> the two stat implementations used in cygwin git, thus:
> 
>   $ ../../test-stat .git/packed-refs
>   stat for '.git/packed-refs':
>   *dev:   -1395862925, 0
>   *ino:   166044, 0
>   *mode:  100644 -rw-, 100600 -rw-
>    nlink: 1, 1
>   *uid:   1005, 0
>   *gid:   513, 0
>   *rdev:  -1395862925, 0
>    size:  296, 296
>    atime: 1371238550, 1371238550 Fri Jun 14 20:35:50 2013
>    mtime: 1371238469, 1371238469 Fri Jun 14 20:34:29 2013
>    ctime: 1371238469, 1371238469 Fri Jun 14 20:34:29 2013

Yikes! ECYGWINFAIL. I have no doubt they have reason why they cannot implement this correctly, but this is rather limiting. (I has assumed that the #ifdef magic that was already in ce_match_stat_basic() would have papered over the problems in stat, but I guess that is not the case.)

You say that there are two stat references in cygwin. Is there a way to ensure that the same one is used in both cases? Or is it so hopelessly broken that there is no point?

Or let me step back and pose a slightly more abstract question:

How, under cygwin, can we implement a quick check of whether a file might have changed? The check should not have any false negatives (claiming that a file is unchanged when actually it was rewritten via the usual lock_file mechanism) nor should it have any "strong" false positives (claiming that a file has changed even though it has never been touched), though a "weak" false positive would be OK (claiming that a file has changed even though it was replaced by a version with identical contents).

If such a check is possible, then we should build it into the implementation of match_stat_data(). If not, we have to think of another way to implement the checks of packed-refs cache up-to-dateness.

(One horrible hack would be: when in doubt, read the packed-refs file into a temporary ref_dir, then compare *the contents* to the version in the cache. If they are the same, then discard the newly-read version, update the stat_validity, and continue to use the old version. If they are different, *then* verify that the lock file was not held, call clear_packed_ref_cache(), and start using the new version.)

Show 21 quoted lines
>   $ ../../bin-wrappers/git -c core.checkstat=minimal pack-refs --all
>   fatal: internal error: packed-ref cache cleared while locked
>   $
> 
> Hmmm, that should have worked! Wait, fix 'git pack-refs' to support
> setting config variables on the command line, rebuild and:
> 
>   $ ../../bin-wrappers/git -c core.checkstat=minimal pack-refs --all
>   $ cat .git/packed-refs
>   # pack-refs with: peeled fully-peeled
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/heads/master
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/outside/foo
>   ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
>   d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e refs/tags/base
>   eb0e854c2cd2511c7571b1a5e3c8b8146193fb30 refs/tags/foo
>   ^d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e
>   $
> 
> I haven't checked the remaining test failures to see if they are
> caused by this code (I don't think so, but ...), but this failure
> is clearly a cygwin specific issue.

Thanks again for the testing and analysis, Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
Ramsay Jones· Jun 18, 2013, 18:13 UTC · re: Michael Haggerty · lore

Re: [PATCH 00/12] Fix some reference-related races

Michael Haggerty wrote:
Show 5 quoted lines
> Thanks for all of the information.
> 
> On 06/15/2013 10:13 PM, Ramsay Jones wrote:
>> Michael Haggerty wrote:
>>> *This patch series must be built on top of mh/reflife.*
[ ... ]
Show 6 quoted lines
>> You may be wondering why clear_packed_ref_cache() is called? Well, that
>> is because stat_validity_check() *incorrectly* indicates that the
>> packed-refs file has changed. Why does it do that? Well, this is one
>> more example of the problems caused by the cygwin schizophrenic stat()
>> functions. :( [ARGHHHHHHHHH]
>>
[ ... ]
> So if I understand correctly, all of the above is *without* the
> refcounting changes introduced in mh/ref-races?  Is so, then it is not
> surprising, as this is exactly the sort of problem that the reference
> counting is meant to solve.

Yes, as I said, this describes the old (non refcounted) series. This particular problem (the segmentation fault) is fixed by the new series (as noted below). [Note, however, that the packed-refs file will still be re-read more often than needed.]

>> Now, turning to the new code, t3211-peel-ref.sh test #7 now works, but
>> test #8 still fails...
[ ... ]
Show 12 quoted lines
> These "internal error: packed-ref cache cleared while locked" failures
> result from an internal consistency check that clear_packed_ref_cache()
> is not called while the write lock is held on the packed-refs file.  A
> call to c_p_r_c() could result from
> 
> * a programming error
> 
> * a determination based on the packed-refs file stat that the file needs
> to be re-read
> 
> Judging from what you said about cygwin, I assume that the latter is
> happening.
Indeed.
>             It should be impossible, because the current process is
> holding packed-refs.lock, and therefore other git processes should
> refuse to change the packed-refs file.
:-P You are assuming that a single process can't lie to itself ...
[ ... ]
> Yikes!  ECYGWINFAIL.

Ah, NO, this should read ECYGWINGITFAIL. This is a self-inflicted wound; it has nothing much to do with cygwin.

I should not have assumed that you knew what I meant by "schizophrenic stat() functions" above; sorry about that! If you are interested, then the following commits may be useful reading: adbc0b6, 7faee6b, 7974843, 05bab3ea, 924aaf3e and b8a97333.

[ ... ]
Show 5 quoted lines
>> I haven't checked the remaining test failures to see if they are
>> caused by this code (I don't think so, but ...), but this failure
>> is clearly a cygwin specific issue.
> 
> Thanks again for the testing and analysis,

So, unless you feel the need to fix this yourself, you can probably ignore this issue for now. I will hopefully find time to fix it up before this topic progresses to next. (Although I don't have any feeling for the time-frame of this topic).

HTH

ATB, Ramsay Jones

Michael Haggerty· Jun 19, 2013, 05:51 UTC · re: Ramsay Jones · lore

Re: [PATCH 00/12] Fix some reference-related races

On 06/18/2013 08:13 PM, Ramsay Jones wrote:
Show 19 quoted lines
> Michael Haggerty wrote:
>> On 06/15/2013 10:13 PM, Ramsay Jones wrote:
>>> Now, turning to the new code, t3211-peel-ref.sh test #7 now works, but
>>> test #8 still fails...
> 
> [ ... ]
> 
>>             It should be impossible, because the current process is
>> holding packed-refs.lock, and therefore other git processes should
>> refuse to change the packed-refs file.
> 
> :-P You are assuming that a single process can't lie to itself ...
> 
> [ ... ]
> 
> I should not have assumed that you knew what I meant by "schizophrenic
> stat() functions" above; sorry about that! If you are interested, then
> the following commits may be useful reading: adbc0b6, 7faee6b, 7974843,
> 05bab3ea, 924aaf3e and b8a97333.
Thanks, that helps.
Show 10 quoted lines
>>> I haven't checked the remaining test failures to see if they are
>>> caused by this code (I don't think so, but ...), but this failure
>>> is clearly a cygwin specific issue.
>>
>> Thanks again for the testing and analysis,
> 
> So, unless you feel the need to fix this yourself, you can probably
> ignore this issue for now. I will hopefully find time to fix it up
> before this topic progresses to next. (Although I don't have any
> feeling for the time-frame of this topic).

Despite reading the commits that you referenced, I still don't feel competent to fix this myself so I gratefully accept your offer. Ideally, whatever complexity is needed would be hidden in the functions stat_validity_check() and stat_validity_update() added by patch 09/12 of my series, and possibly match_stat_data() from 08/12.

Let me know if I can help.
Michael
-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

← back to recent threads