{"thread":{"id":"34217","subject":"[PATCH v2 01/12] repack_without_ref(): split list curation and entry writing","startedAt":"2013-06-19T07:51:21Z","lastAt":"2013-06-20T19:55:26Z","messageCount":24,"participants":["Michael Haggerty","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":12},"messages":[{"id":"221345","messageId":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":null,"subject":"[PATCH v2 00/12] Fix some reference-related races","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:21Z","receivedAt":"2013-06-19T07:51:21Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Re-roll of mh/ref-races.  Thanks to Peff, Junio, and Ramsay for\nreviewing v1.\n\nChanges since v1:\n\n* mh/reflife has graduated to master, so this patch series now applies\n  directly to master.\n\n* Some trivial constness adjustments were necessary because of\n\n    21a6b9fa read-cache: mark cache_entry pointers const\n\n* Better explain (in comments and log message) patch 01/12.\n\n* In 04/12, access struct lock_file::fd directly instead of keeping\n  a separate copy in struct packed_ref_cache.\n\nPlease note that this patch series's usage of stat()/fstat() causes\nbreakages under cygwin, pointed out by Ramsay Jones.  He has\ngraciously offered to work on this issue.\n\nThe last patch is still optional--it avoids a little bit of work when\nrewriting the packed-refs file, but relies on the stat-based freshness\ncheck not giving a false negative.\n\nJeff King (2):\n  get_packed_ref_cache: reload packed-refs file when it changes\n  for_each_ref: load all loose refs before packed refs\n\nMichael Haggerty (10):\n  repack_without_ref(): split list curation and entry writing\n  pack_refs(): split creation of packed refs and entry writing\n  refs: wrap the packed refs cache in a level of indirection\n  refs: implement simple transactions for the packed-refs file\n  refs: manage lifetime of packed refs cache via reference counting\n  do_for_each_entry(): increment the packed refs cache refcount\n  packed_ref_cache: increment refcount when locked\n  Extract a struct stat_data from cache_entry\n  add a stat_validity struct\n  refs: do not invalidate the packed-refs cache unnecessarily\n\n builtin/clone.c    |   7 +-\n builtin/ls-files.c |  12 +-\n cache.h            |  60 ++++++++--\n read-cache.c       | 181 ++++++++++++++++++------------\n refs.c             | 314 ++++++++++++++++++++++++++++++++++++++++++++---------\n refs.h             |  27 ++++-\n 6 files changed, 469 insertions(+), 132 deletions(-)\n\n-- \n1.8.3.1\n"},{"id":"221344","messageId":"1371628293-28824-2-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 01/12] repack_without_ref(): split list curation and entry writing","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:22Z","receivedAt":"2013-06-19T07:51:22Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"The repack_without_ref() function first removes the deleted ref from\nthe internal packed-refs list, then writes the packed-refs list to\ndisk, omitting any broken or stale entries.  This patch splits that\nsecond step into multiple passes:\n\n* collect the list of refnames that should be deleted from packed_refs\n\n* delete those refnames from the cache\n\n* write the remainder to the packed-refs file\n\nThe purpose of this change is to make the \"write the remainder\" part\nreusable.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 50 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 42a7e17..1184b99 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1998,6 +1998,23 @@ static void write_packed_entry(int fd, char *refname, unsigned char *sha1,\n \t}\n }\n \n+/*\n+ * An each_ref_entry_fn that writes the entry to a packed-refs file.\n+ */\n+static int write_packed_entry_fn(struct ref_entry *entry, void *cb_data)\n+{\n+\tint *fd = cb_data;\n+\tenum peel_status peel_status = peel_entry(entry, 0);\n+\n+\tif (peel_status != PEEL_PEELED && peel_status != PEEL_NON_TAG)\n+\t\terror(\"internal error: %s is not a valid packed reference!\",\n+\t\t      entry->name);\n+\twrite_packed_entry(*fd, entry->name, entry->u.value.sha1,\n+\t\t\t   peel_status == PEEL_PEELED ?\n+\t\t\t   entry->u.value.peeled : NULL);\n+\treturn 0;\n+}\n+\n struct ref_to_prune {\n \tstruct ref_to_prune *next;\n \tunsigned char sha1[20];\n@@ -2117,14 +2134,25 @@ int pack_refs(unsigned int flags)\n \treturn 0;\n }\n \n-static int repack_ref_fn(struct ref_entry *entry, void *cb_data)\n+/*\n+ * If entry is no longer needed in packed-refs, add it to the string\n+ * list pointed to by cb_data.  Reasons for deleting entries:\n+ *\n+ * - Entry is broken.\n+ * - Entry is overridden by a loose ref.\n+ * - Entry does not point at a valid object.\n+ *\n+ * In the first and third cases, also emit an error message because these\n+ * are indications of repository corruption.\n+ */\n+static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n {\n-\tint *fd = cb_data;\n-\tenum peel_status peel_status;\n+\tstruct string_list *refs_to_delete = cb_data;\n \n \tif (entry->flag & REF_ISBROKEN) {\n \t\t/* This shouldn't happen to packed refs. */\n \t\terror(\"%s is broken!\", entry->name);\n+\t\tstring_list_append(refs_to_delete, entry->name);\n \t\treturn 0;\n \t}\n \tif (!has_sha1_file(entry->u.value.sha1)) {\n@@ -2134,7 +2162,7 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)\n \t\tif (read_ref_full(entry->name, sha1, 0, &flags))\n \t\t\t/* We should at least have found the packed ref. */\n \t\t\tdie(\"Internal error\");\n-\t\tif ((flags & REF_ISSYMREF) || !(flags & REF_ISPACKED))\n+\t\tif ((flags & REF_ISSYMREF) || !(flags & REF_ISPACKED)) {\n \t\t\t/*\n \t\t\t * This packed reference is overridden by a\n \t\t\t * loose reference, so it is OK that its value\n@@ -2143,9 +2171,11 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)\n \t\t\t * collected.  For this purpose we don't even\n \t\t\t * care whether the loose reference itself is\n \t\t\t * invalid, broken, symbolic, etc.  Silently\n-\t\t\t * omit the packed reference from the output.\n+\t\t\t * remove the packed reference.\n \t\t\t */\n+\t\t\tstring_list_append(refs_to_delete, entry->name);\n \t\t\treturn 0;\n+\t\t}\n \t\t/*\n \t\t * There is no overriding loose reference, so the fact\n \t\t * that this reference doesn't refer to a valid object\n@@ -2154,14 +2184,10 @@ static int repack_ref_fn(struct ref_entry *entry, void *cb_data)\n \t\t * the output.\n \t\t */\n \t\terror(\"%s does not point to a valid object!\", entry->name);\n+\t\tstring_list_append(refs_to_delete, entry->name);\n \t\treturn 0;\n \t}\n \n-\tpeel_status = peel_entry(entry, 0);\n-\twrite_packed_entry(*fd, entry->name, entry->u.value.sha1,\n-\t\t\t   peel_status == PEEL_PEELED ?\n-\t\t\t   entry->u.value.peeled : NULL);\n-\n \treturn 0;\n }\n \n@@ -2169,6 +2195,8 @@ static int repack_without_ref(const char *refname)\n {\n \tint fd;\n \tstruct ref_dir *packed;\n+\tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *ref_to_delete;\n \n \tif (!get_packed_ref(refname))\n \t\treturn 0; /* refname does not exist in packed refs */\n@@ -2180,7 +2208,8 @@ static int repack_without_ref(const char *refname)\n \t}\n \tclear_packed_ref_cache(&ref_cache);\n \tpacked = get_packed_refs(&ref_cache);\n-\t/* Remove refname from the cache. */\n+\n+\t/* Remove refname from the cache: */\n \tif (remove_entry(packed, refname) == -1) {\n \t\t/*\n \t\t * The packed entry disappeared while we were\n@@ -2189,8 +2218,17 @@ static int repack_without_ref(const char *refname)\n \t\trollback_lock_file(&packlock);\n \t\treturn 0;\n \t}\n+\n+\t/* Remove any other accumulated cruft: */\n+\tdo_for_each_entry_in_dir(packed, 0, curate_packed_ref_fn, &refs_to_delete);\n+\tfor_each_string_list_item(ref_to_delete, &refs_to_delete) {\n+\t\tif (remove_entry(packed, ref_to_delete->string) == -1)\n+\t\t\tdie(\"internal error\");\n+\t}\n+\n+\t/* Write what remains: */\n \twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n-\tdo_for_each_entry_in_dir(packed, 0, repack_ref_fn, &fd);\n+\tdo_for_each_entry_in_dir(packed, 0, write_packed_entry_fn, &fd);\n \treturn commit_lock_file(&packlock);\n }\n \n-- \n1.8.3.1\n"},{"id":"221356","messageId":"1371628293-28824-3-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 02/12] pack_refs(): split creation of packed refs and entry writing","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:23Z","receivedAt":"2013-06-19T07:51:23Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Split pack_refs() into multiple passes:\n\n* Iterate over loose refs.  For each one that can be turned into a\n  packed ref, create a corresponding entry in the packed refs cache.\n\n* Write the packed refs to the packed-refs file.\n\nThis change isolates the mutation of the packed-refs file to a single\nplace.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 48 ++++++++++++++++++++++++++++++++++--------------\n 1 file changed, 34 insertions(+), 14 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1184b99..9f1a007 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2023,35 +2023,50 @@ struct ref_to_prune {\n \n struct pack_refs_cb_data {\n \tunsigned int flags;\n+\tstruct ref_dir *packed_refs;\n \tstruct ref_to_prune *ref_to_prune;\n-\tint fd;\n };\n \n-static int pack_one_ref(struct ref_entry *entry, void *cb_data)\n+/*\n+ * An each_ref_entry_fn that is run over loose references only.  If\n+ * the loose reference can be packed, add an entry in the packed ref\n+ * cache.  If the reference should be pruned, also add it to\n+ * ref_to_prune in the pack_refs_cb_data.\n+ */\n+static int pack_if_possible_fn(struct ref_entry *entry, void *cb_data)\n {\n \tstruct pack_refs_cb_data *cb = cb_data;\n \tenum peel_status peel_status;\n+\tstruct ref_entry *packed_entry;\n \tint is_tag_ref = !prefixcmp(entry->name, \"refs/tags/\");\n \n-\t/* ALWAYS pack refs that were already packed or are tags */\n-\tif (!(cb->flags & PACK_REFS_ALL) && !is_tag_ref &&\n-\t    !(entry->flag & REF_ISPACKED))\n+\t/* ALWAYS pack tags */\n+\tif (!(cb->flags & PACK_REFS_ALL) && !is_tag_ref)\n \t\treturn 0;\n \n \t/* Do not pack symbolic or broken refs: */\n \tif ((entry->flag & REF_ISSYMREF) || !ref_resolves_to_object(entry))\n \t\treturn 0;\n \n+\t/* Add a packed ref cache entry equivalent to the loose entry. */\n \tpeel_status = peel_entry(entry, 1);\n \tif (peel_status != PEEL_PEELED && peel_status != PEEL_NON_TAG)\n \t\tdie(\"internal error peeling reference %s (%s)\",\n \t\t    entry->name, sha1_to_hex(entry->u.value.sha1));\n-\twrite_packed_entry(cb->fd, entry->name, entry->u.value.sha1,\n-\t\t\t   peel_status == PEEL_PEELED ?\n-\t\t\t   entry->u.value.peeled : NULL);\n+\tpacked_entry = find_ref(cb->packed_refs, entry->name);\n+\tif (packed_entry) {\n+\t\t/* Overwrite existing packed entry with info from loose entry */\n+\t\tpacked_entry->flag = REF_ISPACKED | REF_KNOWS_PEELED;\n+\t\thashcpy(packed_entry->u.value.sha1, entry->u.value.sha1);\n+\t} else {\n+\t\tpacked_entry = create_ref_entry(entry->name, entry->u.value.sha1,\n+\t\t\t\t\t\tREF_ISPACKED | REF_KNOWS_PEELED, 0);\n+\t\tadd_ref(cb->packed_refs, packed_entry);\n+\t}\n+\thashcpy(packed_entry->u.value.peeled, entry->u.value.peeled);\n \n-\t/* If the ref was already packed, there is no need to prune it. */\n-\tif ((cb->flags & PACK_REFS_PRUNE) && !(entry->flag & REF_ISPACKED)) {\n+\t/* Schedule the loose reference for pruning if requested. */\n+\tif ((cb->flags & PACK_REFS_PRUNE)) {\n \t\tint namelen = strlen(entry->name) + 1;\n \t\tstruct ref_to_prune *n = xcalloc(1, sizeof(*n) + namelen);\n \t\thashcpy(n->sha1, entry->u.value.sha1);\n@@ -2118,16 +2133,21 @@ static struct lock_file packlock;\n int pack_refs(unsigned int flags)\n {\n \tstruct pack_refs_cb_data cbdata;\n+\tint fd;\n \n \tmemset(&cbdata, 0, sizeof(cbdata));\n \tcbdata.flags = flags;\n \n-\tcbdata.fd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"),\n-\t\t\t\t\t      LOCK_DIE_ON_ERROR);\n+\tfd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"),\n+\t\t\t\t       LOCK_DIE_ON_ERROR);\n+\tcbdata.packed_refs = get_packed_refs(&ref_cache);\n \n-\twrite_or_die(cbdata.fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n+\tdo_for_each_entry_in_dir(get_loose_refs(&ref_cache), 0,\n+\t\t\t\t pack_if_possible_fn, &cbdata);\n+\n+\twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n+\tdo_for_each_entry_in_dir(cbdata.packed_refs, 0, write_packed_entry_fn, &fd);\n \n-\tdo_for_each_entry(&ref_cache, \"\", pack_one_ref, &cbdata);\n \tif (commit_lock_file(&packlock) < 0)\n \t\tdie_errno(\"unable to overwrite old ref-pack file\");\n \tprune_refs(cbdata.ref_to_prune);\n-- \n1.8.3.1\n"},{"id":"221353","messageId":"1371628293-28824-4-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 03/12] refs: wrap the packed refs cache in a level of indirection","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:24Z","receivedAt":"2013-06-19T07:51:24Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"As we know, we can solve any problem in this manner.  In this case,\nthe problem is to avoid freeing a packed refs cache while somebody is\nusing it.  So add a level of indirection as a prelude to\nreference-counting the packed refs cache.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 32 ++++++++++++++++++++++++++------\n 1 file changed, 26 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9f1a007..373d95b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -806,6 +806,10 @@ static int is_refname_available(const char *refname, const char *oldrefname,\n \treturn 1;\n }\n \n+struct packed_ref_cache {\n+\tstruct ref_entry *root;\n+};\n+\n /*\n  * Future: need to be in \"struct repository\"\n  * when doing a full libification.\n@@ -813,7 +817,7 @@ static int is_refname_available(const char *refname, const char *oldrefname,\n static struct ref_cache {\n \tstruct ref_cache *next;\n \tstruct ref_entry *loose;\n-\tstruct ref_entry *packed;\n+\tstruct packed_ref_cache *packed;\n \t/*\n \t * The submodule name, or \"\" for the main repo.  We allocate\n \t * length 1 rather than FLEX_ARRAY so that the main ref_cache\n@@ -825,7 +829,8 @@ static struct ref_cache {\n static void clear_packed_ref_cache(struct ref_cache *refs)\n {\n \tif (refs->packed) {\n-\t\tfree_ref_entry(refs->packed);\n+\t\tfree_ref_entry(refs->packed->root);\n+\t\tfree(refs->packed);\n \t\trefs->packed = NULL;\n \t}\n }\n@@ -996,24 +1001,39 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)\n \t}\n }\n \n-static struct ref_dir *get_packed_refs(struct ref_cache *refs)\n+/*\n+ * Get the packed_ref_cache for the specified ref_cache, creating it\n+ * if necessary.\n+ */\n+static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)\n {\n \tif (!refs->packed) {\n \t\tconst char *packed_refs_file;\n \t\tFILE *f;\n \n-\t\trefs->packed = create_dir_entry(refs, \"\", 0, 0);\n+\t\trefs->packed = xcalloc(1, sizeof(*refs->packed));\n+\t\trefs->packed->root = create_dir_entry(refs, \"\", 0, 0);\n \t\tif (*refs->name)\n \t\t\tpacked_refs_file = git_path_submodule(refs->name, \"packed-refs\");\n \t\telse\n \t\t\tpacked_refs_file = git_path(\"packed-refs\");\n \t\tf = fopen(packed_refs_file, \"r\");\n \t\tif (f) {\n-\t\t\tread_packed_refs(f, get_ref_dir(refs->packed));\n+\t\t\tread_packed_refs(f, get_ref_dir(refs->packed->root));\n \t\t\tfclose(f);\n \t\t}\n \t}\n-\treturn get_ref_dir(refs->packed);\n+\treturn refs->packed;\n+}\n+\n+static struct ref_dir *get_packed_ref_dir(struct packed_ref_cache *packed_ref_cache)\n+{\n+\treturn get_ref_dir(packed_ref_cache->root);\n+}\n+\n+static struct ref_dir *get_packed_refs(struct ref_cache *refs)\n+{\n+\treturn get_packed_ref_dir(get_packed_ref_cache(refs));\n }\n \n void add_packed_ref(const char *refname, const unsigned char *sha1)\n-- \n1.8.3.1\n"},{"id":"221347","messageId":"1371628293-28824-5-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:25Z","receivedAt":"2013-06-19T07:51:25Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Handle simple transactions for the packed-refs file at the\npacked_ref_cache level via new functions lock_packed_refs(),\ncommit_packed_refs(), and rollback_packed_refs().\n\nOnly allow the packed ref cache to be modified (via add_packed_ref())\nwhile the packed refs file is locked.\n\nChange clone to add the new references within a transaction.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n builtin/clone.c |  7 ++++-\n refs.c          | 83 ++++++++++++++++++++++++++++++++++++++++++++++-----------\n refs.h          | 27 +++++++++++++++++--\n 3 files changed, 98 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66bff57..b0c000a 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -489,17 +489,22 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n \treturn local_refs;\n }\n \n+static struct lock_file packed_refs_lock;\n+\n static void write_remote_refs(const struct ref *local_refs)\n {\n \tconst struct ref *r;\n \n+\tlock_packed_refs(&packed_refs_lock, LOCK_DIE_ON_ERROR);\n+\n \tfor (r = local_refs; r; r = r->next) {\n \t\tif (!r->peer_ref)\n \t\t\tcontinue;\n \t\tadd_packed_ref(r->peer_ref->name, r->old_sha1);\n \t}\n \n-\tpack_refs(PACK_REFS_ALL);\n+\tif (commit_packed_refs())\n+\t\tdie_errno(\"unable to overwrite old ref-pack file\");\n }\n \n static void write_followtags(const struct ref *refs, const char *msg)\ndiff --git a/refs.c b/refs.c\nindex 373d95b..ad73251 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -808,6 +808,13 @@ static int is_refname_available(const char *refname, const char *oldrefname,\n \n struct packed_ref_cache {\n \tstruct ref_entry *root;\n+\n+\t/*\n+\t * Iff the packed-refs file associated with this instance is\n+\t * currently locked for writing, this points at the associated\n+\t * lock (which is owned by somebody else).\n+\t */\n+\tstruct lock_file *lock;\n };\n \n /*\n@@ -829,6 +836,8 @@ static struct ref_cache {\n static void clear_packed_ref_cache(struct ref_cache *refs)\n {\n \tif (refs->packed) {\n+\t\tif (refs->packed->lock)\n+\t\t\tdie(\"internal error: packed-ref cache cleared while locked\");\n \t\tfree_ref_entry(refs->packed->root);\n \t\tfree(refs->packed);\n \t\trefs->packed = NULL;\n@@ -1038,7 +1047,12 @@ static struct ref_dir *get_packed_refs(struct ref_cache *refs)\n \n void add_packed_ref(const char *refname, const unsigned char *sha1)\n {\n-\tadd_ref(get_packed_refs(&ref_cache),\n+\tstruct packed_ref_cache *packed_ref_cache =\n+\t\tget_packed_ref_cache(&ref_cache);\n+\n+\tif (!packed_ref_cache->lock)\n+\t\tdie(\"internal error: packed refs not locked\");\n+\tadd_ref(get_packed_ref_dir(packed_ref_cache),\n \t\tcreate_ref_entry(refname, sha1, REF_ISPACKED, 1));\n }\n \n@@ -2035,6 +2049,52 @@ static int write_packed_entry_fn(struct ref_entry *entry, void *cb_data)\n \treturn 0;\n }\n \n+int lock_packed_refs(struct lock_file *lock, int flags)\n+{\n+\tstruct packed_ref_cache *packed_ref_cache;\n+\n+\t/* Discard the old cache because it might be invalid: */\n+\tclear_packed_ref_cache(&ref_cache);\n+\tif (hold_lock_file_for_update(lock, git_path(\"packed-refs\"), flags) < 0)\n+\t\treturn -1;\n+\t/* Read the current packed-refs while holding the lock: */\n+\tpacked_ref_cache = get_packed_ref_cache(&ref_cache);\n+\tpacked_ref_cache->lock = lock;\n+\treturn 0;\n+}\n+\n+int commit_packed_refs(void)\n+{\n+\tstruct packed_ref_cache *packed_ref_cache =\n+\t\tget_packed_ref_cache(&ref_cache);\n+\tint error = 0;\n+\n+\tif (!packed_ref_cache->lock)\n+\t\tdie(\"internal error: packed-refs not locked\");\n+\twrite_or_die(packed_ref_cache->lock->fd,\n+\t\t     PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n+\n+\tdo_for_each_entry_in_dir(get_packed_ref_dir(packed_ref_cache),\n+\t\t\t\t 0, write_packed_entry_fn,\n+\t\t\t\t &packed_ref_cache->lock->fd);\n+\tif (commit_lock_file(packed_ref_cache->lock))\n+\t\terror = -1;\n+\tpacked_ref_cache->lock = NULL;\n+\treturn error;\n+}\n+\n+void rollback_packed_refs(void)\n+{\n+\tstruct packed_ref_cache *packed_ref_cache =\n+\t\tget_packed_ref_cache(&ref_cache);\n+\n+\tif (!packed_ref_cache->lock)\n+\t\tdie(\"internal error: packed-refs not locked\");\n+\trollback_lock_file(packed_ref_cache->lock);\n+\tpacked_ref_cache->lock = NULL;\n+\tclear_packed_ref_cache(&ref_cache);\n+}\n+\n struct ref_to_prune {\n \tstruct ref_to_prune *next;\n \tunsigned char sha1[20];\n@@ -2153,23 +2213,19 @@ static struct lock_file packlock;\n int pack_refs(unsigned int flags)\n {\n \tstruct pack_refs_cb_data cbdata;\n-\tint fd;\n \n \tmemset(&cbdata, 0, sizeof(cbdata));\n \tcbdata.flags = flags;\n \n-\tfd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"),\n-\t\t\t\t       LOCK_DIE_ON_ERROR);\n+\tlock_packed_refs(&packlock, LOCK_DIE_ON_ERROR);\n \tcbdata.packed_refs = get_packed_refs(&ref_cache);\n \n \tdo_for_each_entry_in_dir(get_loose_refs(&ref_cache), 0,\n \t\t\t\t pack_if_possible_fn, &cbdata);\n \n-\twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n-\tdo_for_each_entry_in_dir(cbdata.packed_refs, 0, write_packed_entry_fn, &fd);\n-\n-\tif (commit_lock_file(&packlock) < 0)\n+\tif (commit_packed_refs())\n \t\tdie_errno(\"unable to overwrite old ref-pack file\");\n+\n \tprune_refs(cbdata.ref_to_prune);\n \treturn 0;\n }\n@@ -2233,7 +2289,6 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n \n static int repack_without_ref(const char *refname)\n {\n-\tint fd;\n \tstruct ref_dir *packed;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *ref_to_delete;\n@@ -2241,12 +2296,10 @@ static int repack_without_ref(const char *refname)\n \tif (!get_packed_ref(refname))\n \t\treturn 0; /* refname does not exist in packed refs */\n \n-\tfd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"), 0);\n-\tif (fd < 0) {\n+\tif (lock_packed_refs(&packlock, 0)) {\n \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n \t\treturn error(\"cannot delete '%s' from packed refs\", refname);\n \t}\n-\tclear_packed_ref_cache(&ref_cache);\n \tpacked = get_packed_refs(&ref_cache);\n \n \t/* Remove refname from the cache: */\n@@ -2255,7 +2308,7 @@ static int repack_without_ref(const char *refname)\n \t\t * The packed entry disappeared while we were\n \t\t * acquiring the lock.\n \t\t */\n-\t\trollback_lock_file(&packlock);\n+\t\trollback_packed_refs();\n \t\treturn 0;\n \t}\n \n@@ -2267,9 +2320,7 @@ static int repack_without_ref(const char *refname)\n \t}\n \n \t/* Write what remains: */\n-\twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n-\tdo_for_each_entry_in_dir(packed, 0, write_packed_entry_fn, &fd);\n-\treturn commit_lock_file(&packlock);\n+\treturn commit_packed_refs();\n }\n \n int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\ndiff --git a/refs.h b/refs.h\nindex 246bf60..b275124 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -77,12 +77,35 @@ extern int for_each_rawref(each_ref_fn, void *);\n extern void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname);\n \n /*\n- * Add a reference to the in-memory packed reference cache.  To actually\n- * write the reference to the packed-refs file, call pack_refs().\n+ * Lock the packed-refs file for writing using the specified lock_file\n+ * instance.  Flags is passed to hold_lock_file_for_update().  Return\n+ * 0 on success.\n+ */\n+extern int lock_packed_refs(struct lock_file *lock, int flags);\n+\n+/*\n+ * Add a reference to the in-memory packed reference cache.  This may\n+ * only be called while the packed-refs file is locked (see\n+ * lock_packed_refs()).  To actually write the packed-refs file, call\n+ * commit_packed_refs().\n  */\n extern void add_packed_ref(const char *refname, const unsigned char *sha1);\n \n /*\n+ * Write the current version of the packed refs cache from memory to\n+ * disk.  The packed-refs file must already be locked for writing (see\n+ * lock_packed_refs()).  Return zero on success.\n+ */\n+extern int commit_packed_refs(void);\n+\n+/*\n+ * Rollback the lockfile for the packed-refs file, and discard the\n+ * in-memory packed reference cache.  (The packed-refs file will be\n+ * read anew if it is needed again after this function is called.)\n+ */\n+extern void rollback_packed_refs(void);\n+\n+/*\n  * Flags for controlling behaviour of pack_refs()\n  * PACK_REFS_PRUNE: Prune loose refs after packing\n  * PACK_REFS_ALL:   Pack _all_ refs, not just tags and already packed refs\n-- \n1.8.3.1\n"},{"id":"221346","messageId":"1371628293-28824-6-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 05/12] refs: manage lifetime of packed refs cache via reference counting","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:26Z","receivedAt":"2013-06-19T07:51:26Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"In struct packed_ref_cache, keep a count of the number of users of the\ndata structure.  Only free the packed ref cache when the reference\ncount goes to zero rather than when the packed ref cache is cleared.\nThis mechanism will be used to prevent the cache data structure from\nbeing freed while it is being iterated over.\n\nSo far, only the reference in struct ref_cache::packed is counted;\nother users will be adjusted in separate commits.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 39 ++++++++++++++++++++++++++++++++++++---\n 1 file changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex ad73251..e2e441d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -810,6 +810,14 @@ struct packed_ref_cache {\n \tstruct ref_entry *root;\n \n \t/*\n+\t * Count of references to the data structure in this instance,\n+\t * including the pointer from ref_cache::packed if any.  The\n+\t * data will not be freed as long as the reference count is\n+\t * nonzero.\n+\t */\n+\tunsigned int referrers;\n+\n+\t/*\n \t * Iff the packed-refs file associated with this instance is\n \t * currently locked for writing, this points at the associated\n \t * lock (which is owned by somebody else).\n@@ -833,14 +841,38 @@ static struct ref_cache {\n \tchar name[1];\n } ref_cache, *submodule_ref_caches;\n \n+/*\n+ * Increment the reference count of *packed_refs.\n+ */\n+static void acquire_packed_ref_cache(struct packed_ref_cache *packed_refs)\n+{\n+\tpacked_refs->referrers++;\n+}\n+\n+/*\n+ * Decrease the reference count of *packed_refs.  If it goes to zero,\n+ * free *packed_refs and return true; otherwise return false.\n+ */\n+static int release_packed_ref_cache(struct packed_ref_cache *packed_refs)\n+{\n+\tif (!--packed_refs->referrers) {\n+\t\tfree_ref_entry(packed_refs->root);\n+\t\tfree(packed_refs);\n+\t\treturn 1;\n+\t} else {\n+\t\treturn 0;\n+\t}\n+}\n+\n static void clear_packed_ref_cache(struct ref_cache *refs)\n {\n \tif (refs->packed) {\n-\t\tif (refs->packed->lock)\n+\t\tstruct packed_ref_cache *packed_refs = refs->packed;\n+\n+\t\tif (packed_refs->lock)\n \t\t\tdie(\"internal error: packed-ref cache cleared while locked\");\n-\t\tfree_ref_entry(refs->packed->root);\n-\t\tfree(refs->packed);\n \t\trefs->packed = NULL;\n+\t\trelease_packed_ref_cache(packed_refs);\n \t}\n }\n \n@@ -1021,6 +1053,7 @@ static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)\n \t\tFILE *f;\n \n \t\trefs->packed = xcalloc(1, sizeof(*refs->packed));\n+\t\tacquire_packed_ref_cache(refs->packed);\n \t\trefs->packed->root = create_dir_entry(refs, \"\", 0, 0);\n \t\tif (*refs->name)\n \t\t\tpacked_refs_file = git_path_submodule(refs->name, \"packed-refs\");\n-- \n1.8.3.1\n"},{"id":"221349","messageId":"1371628293-28824-7-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 06/12] do_for_each_entry(): increment the packed refs cache refcount","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:27Z","receivedAt":"2013-06-19T07:51:27Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"This function calls a user-supplied callback function which could do\nsomething that causes the packed refs cache to be invalidated.  So\nacquire a reference count on the data structure to prevent our copy\nfrom being freed while we are iterating over it.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex e2e441d..d90f487 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1587,10 +1587,12 @@ void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)\n static int do_for_each_entry(struct ref_cache *refs, const char *base,\n \t\t\t     each_ref_entry_fn fn, void *cb_data)\n {\n-\tstruct ref_dir *packed_dir = get_packed_refs(refs);\n+\tstruct packed_ref_cache *packed_ref_cache = get_packed_ref_cache(refs);\n+\tstruct ref_dir *packed_dir = get_packed_ref_dir(packed_ref_cache);\n \tstruct ref_dir *loose_dir = get_loose_refs(refs);\n \tint retval = 0;\n \n+\tacquire_packed_ref_cache(packed_ref_cache);\n \tif (base && *base) {\n \t\tpacked_dir = find_containing_dir(packed_dir, base, 0);\n \t\tloose_dir = find_containing_dir(loose_dir, base, 0);\n@@ -1611,6 +1613,7 @@ static int do_for_each_entry(struct ref_cache *refs, const char *base,\n \t\t\t\tloose_dir, 0, fn, cb_data);\n \t}\n \n+\trelease_packed_ref_cache(packed_ref_cache);\n \treturn retval;\n }\n \n-- \n1.8.3.1\n"},{"id":"221352","messageId":"1371628293-28824-8-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 07/12] packed_ref_cache: increment refcount when locked","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:28Z","receivedAt":"2013-06-19T07:51:28Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Increment the packed_ref_cache reference count while it is locked to\nprevent its being freed.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex d90f487..6d48b42 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -820,7 +820,9 @@ struct packed_ref_cache {\n \t/*\n \t * Iff the packed-refs file associated with this instance is\n \t * currently locked for writing, this points at the associated\n-\t * lock (which is owned by somebody else).\n+\t * lock (which is owned by somebody else).  The referrer count\n+\t * is also incremented when the file is locked and decremented\n+\t * when it is unlocked.\n \t */\n \tstruct lock_file *lock;\n };\n@@ -2096,6 +2098,8 @@ int lock_packed_refs(struct lock_file *lock, int flags)\n \t/* Read the current packed-refs while holding the lock: */\n \tpacked_ref_cache = get_packed_ref_cache(&ref_cache);\n \tpacked_ref_cache->lock = lock;\n+\t/* Increment the reference count to prevent it from being freed: */\n+\tacquire_packed_ref_cache(packed_ref_cache);\n \treturn 0;\n }\n \n@@ -2116,6 +2120,7 @@ int commit_packed_refs(void)\n \tif (commit_lock_file(packed_ref_cache->lock))\n \t\terror = -1;\n \tpacked_ref_cache->lock = NULL;\n+\trelease_packed_ref_cache(packed_ref_cache);\n \treturn error;\n }\n \n@@ -2128,6 +2133,7 @@ void rollback_packed_refs(void)\n \t\tdie(\"internal error: packed-refs not locked\");\n \trollback_lock_file(packed_ref_cache->lock);\n \tpacked_ref_cache->lock = NULL;\n+\trelease_packed_ref_cache(packed_ref_cache);\n \tclear_packed_ref_cache(&ref_cache);\n }\n \n-- \n1.8.3.1\n"},{"id":"221355","messageId":"1371628293-28824-9-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 08/12] Extract a struct stat_data from cache_entry","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:29Z","receivedAt":"2013-06-19T07:51:29Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Add public functions fill_stat_data() and match_stat_data() to work\nwith it.  This infrastructure will later be used to check the validity\nof other types of file.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n builtin/ls-files.c |  12 +++--\n cache.h            |  33 +++++++++---\n read-cache.c       | 151 +++++++++++++++++++++++++++++------------------------\n 3 files changed, 116 insertions(+), 80 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 2202072..6a0730f 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -165,11 +165,13 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)\n \t}\n \twrite_name(ce->name, ce_namelen(ce));\n \tif (debug_mode) {\n-\t\tprintf(\"  ctime: %d:%d\\n\", ce->ce_ctime.sec, ce->ce_ctime.nsec);\n-\t\tprintf(\"  mtime: %d:%d\\n\", ce->ce_mtime.sec, ce->ce_mtime.nsec);\n-\t\tprintf(\"  dev: %d\\tino: %d\\n\", ce->ce_dev, ce->ce_ino);\n-\t\tprintf(\"  uid: %d\\tgid: %d\\n\", ce->ce_uid, ce->ce_gid);\n-\t\tprintf(\"  size: %d\\tflags: %x\\n\", ce->ce_size, ce->ce_flags);\n+\t\tstruct stat_data *sd = &ce->ce_stat_data;\n+\n+\t\tprintf(\"  ctime: %d:%d\\n\", sd->sd_ctime.sec, sd->sd_ctime.nsec);\n+\t\tprintf(\"  mtime: %d:%d\\n\", sd->sd_mtime.sec, sd->sd_mtime.nsec);\n+\t\tprintf(\"  dev: %d\\tino: %d\\n\", sd->sd_dev, sd->sd_ino);\n+\t\tprintf(\"  uid: %d\\tgid: %d\\n\", sd->sd_uid, sd->sd_gid);\n+\t\tprintf(\"  size: %d\\tflags: %x\\n\", sd->sd_size, ce->ce_flags);\n \t}\n }\n \ndiff --git a/cache.h b/cache.h\nindex 820aa05..f158fed 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -119,15 +119,19 @@ struct cache_time {\n \tunsigned int nsec;\n };\n \n+struct stat_data {\n+\tstruct cache_time sd_ctime;\n+\tstruct cache_time sd_mtime;\n+\tunsigned int sd_dev;\n+\tunsigned int sd_ino;\n+\tunsigned int sd_uid;\n+\tunsigned int sd_gid;\n+\tunsigned int sd_size;\n+};\n+\n struct cache_entry {\n-\tstruct cache_time ce_ctime;\n-\tstruct cache_time ce_mtime;\n-\tunsigned int ce_dev;\n-\tunsigned int ce_ino;\n+\tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n-\tunsigned int ce_uid;\n-\tunsigned int ce_gid;\n-\tunsigned int ce_size;\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n \tunsigned char sha1[20];\n@@ -511,6 +515,21 @@ extern int limit_pathspec_to_literal(void);\n #define HASH_FORMAT_CHECK 2\n extern int index_fd(unsigned char *sha1, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n extern int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags);\n+\n+/*\n+ * Record to sd the data from st that we use to check whether a file\n+ * might have changed.\n+ */\n+extern void fill_stat_data(struct stat_data *sd, struct stat *st);\n+\n+/*\n+ * Return 0 if st is consistent with a file not having been changed\n+ * since sd was filled.  If there are differences, return a\n+ * combination of MTIME_CHANGED, CTIME_CHANGED, OWNER_CHANGED,\n+ * INODE_CHANGED, and DATA_CHANGED.\n+ */\n+extern int match_stat_data(const struct stat_data *sd, struct stat *st);\n+\n extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n \n #define REFRESH_REALLY\t\t0x0001\t/* ignore_valid */\ndiff --git a/read-cache.c b/read-cache.c\nindex 5e30746..5660b37 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -67,6 +67,61 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n \n+void fill_stat_data(struct stat_data *sd, struct stat *st)\n+{\n+\tsd->sd_ctime.sec = (unsigned int)st->st_ctime;\n+\tsd->sd_mtime.sec = (unsigned int)st->st_mtime;\n+\tsd->sd_ctime.nsec = ST_CTIME_NSEC(*st);\n+\tsd->sd_mtime.nsec = ST_MTIME_NSEC(*st);\n+\tsd->sd_dev = st->st_dev;\n+\tsd->sd_ino = st->st_ino;\n+\tsd->sd_uid = st->st_uid;\n+\tsd->sd_gid = st->st_gid;\n+\tsd->sd_size = st->st_size;\n+}\n+\n+int match_stat_data(const struct stat_data *sd, struct stat *st)\n+{\n+\tint changed = 0;\n+\n+\tif (sd->sd_mtime.sec != (unsigned int)st->st_mtime)\n+\t\tchanged |= MTIME_CHANGED;\n+\tif (trust_ctime && check_stat &&\n+\t    sd->sd_ctime.sec != (unsigned int)st->st_ctime)\n+\t\tchanged |= CTIME_CHANGED;\n+\n+#ifdef USE_NSEC\n+\tif (check_stat && sd->sd_mtime.nsec != ST_MTIME_NSEC(*st))\n+\t\tchanged |= MTIME_CHANGED;\n+\tif (trust_ctime && check_stat &&\n+\t    sd->sd_ctime.nsec != ST_CTIME_NSEC(*st))\n+\t\tchanged |= CTIME_CHANGED;\n+#endif\n+\n+\tif (check_stat) {\n+\t\tif (sd->sd_uid != (unsigned int) st->st_uid ||\n+\t\t\tsd->sd_gid != (unsigned int) st->st_gid)\n+\t\t\tchanged |= OWNER_CHANGED;\n+\t\tif (sd->sd_ino != (unsigned int) st->st_ino)\n+\t\t\tchanged |= INODE_CHANGED;\n+\t}\n+\n+#ifdef USE_STDEV\n+\t/*\n+\t * st_dev breaks on network filesystems where different\n+\t * clients will have different views of what \"device\"\n+\t * the filesystem is on\n+\t */\n+\tif (check_stat && sd->sd_dev != (unsigned int) st->st_dev)\n+\t\t\tchanged |= INODE_CHANGED;\n+#endif\n+\n+\tif (sd->sd_size != (unsigned int) st->st_size)\n+\t\tchanged |= DATA_CHANGED;\n+\n+\treturn changed;\n+}\n+\n /*\n  * This only updates the \"non-critical\" parts of the directory\n  * cache, ie the parts that aren't tracked by GIT, and only used\n@@ -74,15 +129,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n  */\n void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n {\n-\tce->ce_ctime.sec = (unsigned int)st->st_ctime;\n-\tce->ce_mtime.sec = (unsigned int)st->st_mtime;\n-\tce->ce_ctime.nsec = ST_CTIME_NSEC(*st);\n-\tce->ce_mtime.nsec = ST_MTIME_NSEC(*st);\n-\tce->ce_dev = st->st_dev;\n-\tce->ce_ino = st->st_ino;\n-\tce->ce_uid = st->st_uid;\n-\tce->ce_gid = st->st_gid;\n-\tce->ce_size = st->st_size;\n+\tfill_stat_data(&ce->ce_stat_data, st);\n \n \tif (assume_unchanged)\n \t\tce->ce_flags |= CE_VALID;\n@@ -195,43 +242,11 @@ static int ce_match_stat_basic(const struct cache_entry *ce, struct stat *st)\n \tdefault:\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n-\tif (ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n-\t\tchanged |= MTIME_CHANGED;\n-\tif (trust_ctime && check_stat &&\n-\t    ce->ce_ctime.sec != (unsigned int)st->st_ctime)\n-\t\tchanged |= CTIME_CHANGED;\n \n-#ifdef USE_NSEC\n-\tif (check_stat && ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))\n-\t\tchanged |= MTIME_CHANGED;\n-\tif (trust_ctime && check_stat &&\n-\t    ce->ce_ctime.nsec != ST_CTIME_NSEC(*st))\n-\t\tchanged |= CTIME_CHANGED;\n-#endif\n-\n-\tif (check_stat) {\n-\t\tif (ce->ce_uid != (unsigned int) st->st_uid ||\n-\t\t\tce->ce_gid != (unsigned int) st->st_gid)\n-\t\t\tchanged |= OWNER_CHANGED;\n-\t\tif (ce->ce_ino != (unsigned int) st->st_ino)\n-\t\t\tchanged |= INODE_CHANGED;\n-\t}\n-\n-#ifdef USE_STDEV\n-\t/*\n-\t * st_dev breaks on network filesystems where different\n-\t * clients will have different views of what \"device\"\n-\t * the filesystem is on\n-\t */\n-\tif (check_stat && ce->ce_dev != (unsigned int) st->st_dev)\n-\t\t\tchanged |= INODE_CHANGED;\n-#endif\n-\n-\tif (ce->ce_size != (unsigned int) st->st_size)\n-\t\tchanged |= DATA_CHANGED;\n+\tchanged |= match_stat_data(&ce->ce_stat_data, st);\n \n \t/* Racily smudged entry? */\n-\tif (!ce->ce_size) {\n+\tif (!ce->ce_stat_data.sd_size) {\n \t\tif (!is_empty_blob_sha1(ce->sha1))\n \t\t\tchanged |= DATA_CHANGED;\n \t}\n@@ -246,11 +261,11 @@ static int is_racy_timestamp(const struct index_state *istate,\n \t\tistate->timestamp.sec &&\n #ifdef USE_NSEC\n \t\t /* nanosecond timestamped files can also be racy! */\n-\t\t(istate->timestamp.sec < ce->ce_mtime.sec ||\n-\t\t (istate->timestamp.sec == ce->ce_mtime.sec &&\n-\t\t  istate->timestamp.nsec <= ce->ce_mtime.nsec))\n+\t\t(istate->timestamp.sec < ce->ce_stat_data.sd_mtime.sec ||\n+\t\t (istate->timestamp.sec == ce->ce_stat_data.sd_mtime.sec &&\n+\t\t  istate->timestamp.nsec <= ce->ce_stat_data.sd_mtime.nsec))\n #else\n-\t\tistate->timestamp.sec <= ce->ce_mtime.sec\n+\t\tistate->timestamp.sec <= ce->ce_stat_data.sd_mtime.sec\n #endif\n \t\t );\n }\n@@ -342,7 +357,7 @@ int ie_modified(const struct index_state *istate,\n \t * then we know it is.\n \t */\n \tif ((changed & DATA_CHANGED) &&\n-\t    (S_ISGITLINK(ce->ce_mode) || ce->ce_size != 0))\n+\t    (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0))\n \t\treturn changed;\n \n \tchanged_fs = ce_modified_check_fs(ce, st);\n@@ -1324,16 +1339,16 @@ static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *on\n {\n \tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n \n-\tce->ce_ctime.sec = ntoh_l(ondisk->ctime.sec);\n-\tce->ce_mtime.sec = ntoh_l(ondisk->mtime.sec);\n-\tce->ce_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n-\tce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n-\tce->ce_dev   = ntoh_l(ondisk->dev);\n-\tce->ce_ino   = ntoh_l(ondisk->ino);\n+\tce->ce_stat_data.sd_ctime.sec = ntoh_l(ondisk->ctime.sec);\n+\tce->ce_stat_data.sd_mtime.sec = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_stat_data.sd_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n+\tce->ce_stat_data.sd_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_stat_data.sd_dev   = ntoh_l(ondisk->dev);\n+\tce->ce_stat_data.sd_ino   = ntoh_l(ondisk->ino);\n \tce->ce_mode  = ntoh_l(ondisk->mode);\n-\tce->ce_uid   = ntoh_l(ondisk->uid);\n-\tce->ce_gid   = ntoh_l(ondisk->gid);\n-\tce->ce_size  = ntoh_l(ondisk->size);\n+\tce->ce_stat_data.sd_uid   = ntoh_l(ondisk->uid);\n+\tce->ce_stat_data.sd_gid   = ntoh_l(ondisk->gid);\n+\tce->ce_stat_data.sd_size  = ntoh_l(ondisk->size);\n \tce->ce_flags = flags & ~CE_NAMEMASK;\n \tce->ce_namelen = len;\n \thashcpy(ce->sha1, ondisk->sha1);\n@@ -1610,7 +1625,7 @@ static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n \t * The only thing we care about in this function is to smudge the\n \t * falsely clean entry due to touch-update-touch race, so we leave\n \t * everything else as they are.  We are called for entries whose\n-\t * ce_mtime match the index file mtime.\n+\t * ce_stat_data.sd_mtime match the index file mtime.\n \t *\n \t * Note that this actually does not do much for gitlinks, for\n \t * which ce_match_stat_basic() always goes to the actual\n@@ -1649,7 +1664,7 @@ static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n \t\t * file, and never calls us, so the cached size information\n \t\t * for \"frotz\" stays 6 which does not match the filesystem.\n \t\t */\n-\t\tce->ce_size = 0;\n+\t\tce->ce_stat_data.sd_size = 0;\n \t}\n }\n \n@@ -1659,16 +1674,16 @@ static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n {\n \tshort flags;\n \n-\tondisk->ctime.sec = htonl(ce->ce_ctime.sec);\n-\tondisk->mtime.sec = htonl(ce->ce_mtime.sec);\n-\tondisk->ctime.nsec = htonl(ce->ce_ctime.nsec);\n-\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n-\tondisk->dev  = htonl(ce->ce_dev);\n-\tondisk->ino  = htonl(ce->ce_ino);\n+\tondisk->ctime.sec = htonl(ce->ce_stat_data.sd_ctime.sec);\n+\tondisk->mtime.sec = htonl(ce->ce_stat_data.sd_mtime.sec);\n+\tondisk->ctime.nsec = htonl(ce->ce_stat_data.sd_ctime.nsec);\n+\tondisk->mtime.nsec = htonl(ce->ce_stat_data.sd_mtime.nsec);\n+\tondisk->dev  = htonl(ce->ce_stat_data.sd_dev);\n+\tondisk->ino  = htonl(ce->ce_stat_data.sd_ino);\n \tondisk->mode = htonl(ce->ce_mode);\n-\tondisk->uid  = htonl(ce->ce_uid);\n-\tondisk->gid  = htonl(ce->ce_gid);\n-\tondisk->size = htonl(ce->ce_size);\n+\tondisk->uid  = htonl(ce->ce_stat_data.sd_uid);\n+\tondisk->gid  = htonl(ce->ce_stat_data.sd_gid);\n+\tondisk->size = htonl(ce->ce_stat_data.sd_size);\n \thashcpy(ondisk->sha1, ce->sha1);\n \n \tflags = ce->ce_flags;\n-- \n1.8.3.1\n"},{"id":"221348","messageId":"1371628293-28824-10-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 09/12] add a stat_validity struct","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:30Z","receivedAt":"2013-06-19T07:51:30Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"It can sometimes be useful to know whether a path in the\nfilesystem has been updated without going to the work of\nopening and re-reading its content. We trust the stat()\ninformation on disk already to handle index updates, and we\ncan use the same trick here.\n\nThis patch introduces a \"stat_validity\" struct which\nencapsulates the concept of checking the stat-freshness of a\nfile. It is implemented on top of \"struct stat_data\" to\nreuse the logic about which stat entries to trust for a\nparticular platform, but hides the complexity behind two\nsimple functions: check and update.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n cache.h      | 27 +++++++++++++++++++++++++++\n read-cache.c | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex f158fed..50f33db 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1360,4 +1360,31 @@ int checkout_fast_forward(const unsigned char *from,\n \n int sane_execvp(const char *file, char *const argv[]);\n \n+/*\n+ * A struct to encapsulate the concept of whether a file has changed\n+ * since we last checked it. This uses criteria similar to those used\n+ * for the index.\n+ */\n+struct stat_validity {\n+\tstruct stat_data *sd;\n+};\n+\n+void stat_validity_clear(struct stat_validity *sv);\n+\n+/*\n+ * Returns 1 if the path is a regular file (or a symlink to a regular\n+ * file) and matches the saved stat_validity, 0 otherwise.  A missing\n+ * or inaccessible file is considered a match if the struct was just\n+ * initialized, or if the previous update found an inaccessible file.\n+ */\n+int stat_validity_check(struct stat_validity *sv, const char *path);\n+\n+/*\n+ * Update the stat_validity from a file opened at descriptor fd. If\n+ * the file is missing, inaccessible, or not a regular file, then\n+ * future calls to stat_validity_check will match iff one of those\n+ * conditions continues to be true.\n+ */\n+void stat_validity_update(struct stat_validity *sv, int fd);\n+\n #endif /* CACHE_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex 5660b37..b15bc09 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1950,3 +1950,33 @@ void *read_blob_data_from_index(struct index_state *istate, const char *path, un\n \t\t*size = sz;\n \treturn data;\n }\n+\n+void stat_validity_clear(struct stat_validity *sv)\n+{\n+\tfree(sv->sd);\n+\tsv->sd = NULL;\n+}\n+\n+int stat_validity_check(struct stat_validity *sv, const char *path)\n+{\n+\tstruct stat st;\n+\n+\tif (stat(path, &st) < 0)\n+\t\treturn sv->sd == NULL;\n+\tif (!sv->sd)\n+\t\treturn 0;\n+\treturn S_ISREG(st.st_mode) && !match_stat_data(sv->sd, &st);\n+}\n+\n+void stat_validity_update(struct stat_validity *sv, int fd)\n+{\n+\tstruct stat st;\n+\n+\tif (fstat(fd, &st) < 0 || !S_ISREG(st.st_mode))\n+\t\tstat_validity_clear(sv);\n+\telse {\n+\t\tif (!sv->sd)\n+\t\t\tsv->sd = xcalloc(1, sizeof(struct stat_data));\n+\t\tfill_stat_data(sv->sd, &st);\n+\t}\n+}\n-- \n1.8.3.1\n"},{"id":"221354","messageId":"1371628293-28824-11-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 10/12] get_packed_ref_cache: reload packed-refs file when it changes","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:31Z","receivedAt":"2013-06-19T07:51:31Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nOnce we read the packed-refs file into memory, we cache it\nto save work on future ref lookups. However, our cache may\nbe out of date with respect to what is on disk if another\nprocess is simultaneously packing the refs. Normally it\nis acceptable for us to be a little out of date, since there\nis no guarantee whether we read the file before or after the\nsimultaneous update. However, there is an important special\ncase: our packed-refs file must be up to date with respect\nto any loose refs we read. Otherwise, we risk the following\nrace condition:\n\n  0. There exists a loose ref refs/heads/master.\n\n  1. Process A starts and looks up the ref \"master\". It\n     first checks $GIT_DIR/master, which does not exist. It\n     then loads (and caches) the packed-refs file to see if\n     \"master\" exists in it, which it does not.\n\n  2. Meanwhile, process B runs \"pack-refs --all --prune\". It\n     creates a new packed-refs file which contains\n     refs/heads/master, and removes the loose copy at\n     $GIT_DIR/refs/heads/master.\n\n  3. Process A continues its lookup, and eventually tries\n     $GIT_DIR/refs/heads/master.  It sees that the loose ref\n     is missing, and falls back to the packed-refs file. But\n     it examines its cached version, which does not have\n     refs/heads/master. After trying a few other prefixes,\n     it reports master as a non-existent ref.\n\nThere are many variants (e.g., step 1 may involve process A\nlooking up another ref entirely, so even a fully qualified\nrefname can fail). One of the most interesting ones is if\n\"refs/heads/master\" is already packed. In that case process\nA will not see it as missing, but rather will report\nwhatever value happened to be in the packed-refs file before\nprocess B repacked (which might be an arbitrarily old\nvalue).\n\nWe can fix this by making sure we reload the packed-refs\nfile from disk after looking at any loose refs. That's\nunacceptably slow, so we can check its stat()-validity as a\nproxy, and read it only when it appears to have changed.\n\nReading the packed-refs file after performing any loose-ref\nsystem calls is sufficient because we know the ordering of\nthe pack-refs process: it always makes sure the newly\nwritten packed-refs file is installed into place before\npruning any loose refs. As long as those operations by B\nappear in their executed order to process A, by the time A\nsees the missing loose ref, the new packed-refs file must be\nin place.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 21 ++++++++++++++++-----\n 1 file changed, 16 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 6d48b42..d53a172 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -825,6 +825,9 @@ struct packed_ref_cache {\n \t * when it is unlocked.\n \t */\n \tstruct lock_file *lock;\n+\n+\t/* The metadata from when this packed-refs cache was read */\n+\tstruct stat_validity validity;\n };\n \n /*\n@@ -859,6 +862,7 @@ static int release_packed_ref_cache(struct packed_ref_cache *packed_refs)\n {\n \tif (!--packed_refs->referrers) {\n \t\tfree_ref_entry(packed_refs->root);\n+\t\tstat_validity_clear(&packed_refs->validity);\n \t\tfree(packed_refs);\n \t\treturn 1;\n \t} else {\n@@ -1050,19 +1054,26 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)\n  */\n static struct packed_ref_cache *get_packed_ref_cache(struct ref_cache *refs)\n {\n+\tconst char *packed_refs_file;\n+\n+\tif (*refs->name)\n+\t\tpacked_refs_file = git_path_submodule(refs->name, \"packed-refs\");\n+\telse\n+\t\tpacked_refs_file = git_path(\"packed-refs\");\n+\n+\tif (refs->packed &&\n+\t    !stat_validity_check(&refs->packed->validity, packed_refs_file))\n+\t\tclear_packed_ref_cache(refs);\n+\n \tif (!refs->packed) {\n-\t\tconst char *packed_refs_file;\n \t\tFILE *f;\n \n \t\trefs->packed = xcalloc(1, sizeof(*refs->packed));\n \t\tacquire_packed_ref_cache(refs->packed);\n \t\trefs->packed->root = create_dir_entry(refs, \"\", 0, 0);\n-\t\tif (*refs->name)\n-\t\t\tpacked_refs_file = git_path_submodule(refs->name, \"packed-refs\");\n-\t\telse\n-\t\t\tpacked_refs_file = git_path(\"packed-refs\");\n \t\tf = fopen(packed_refs_file, \"r\");\n \t\tif (f) {\n+\t\t\tstat_validity_update(&refs->packed->validity, fileno(f));\n \t\t\tread_packed_refs(f, get_ref_dir(refs->packed->root));\n \t\t\tfclose(f);\n \t\t}\n-- \n1.8.3.1\n"},{"id":"221350","messageId":"1371628293-28824-12-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 11/12] for_each_ref: load all loose refs before packed refs","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:32Z","receivedAt":"2013-06-19T07:51:32Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIf we are iterating through the refs using for_each_ref (or\nany of its sister functions), we can get into a race\ncondition with a simultaneous \"pack-refs --prune\" that looks\nlike this:\n\n  0. We have a large number of loose refs, and a few packed\n     refs. refs/heads/z/foo is loose, with no matching entry\n     in the packed-refs file.\n\n  1. Process A starts iterating through the refs. It loads\n     the packed-refs file from disk, then starts lazily\n     traversing through the loose ref directories.\n\n  2. Process B, running \"pack-refs --prune\", writes out the\n     new packed-refs file. It then deletes the newly packed\n     refs, including refs/heads/z/foo.\n\n  3. Meanwhile, process A has finally gotten to\n     refs/heads/z (it traverses alphabetically). It\n     descends, but finds nothing there.  It checks its\n     cached view of the packed-refs file, but it does not\n     mention anything in \"refs/heads/z/\" at all (it predates\n     the new file written by B in step 2).\n\nThe traversal completes successfully without mentioning\nrefs/heads/z/foo at all (the name, of course, isn't\nimportant; but the more refs you have and the farther down\nthe alphabetical list a ref is, the more likely it is to hit\nthe race). If refs/heads/z/foo did exist in the packed refs\nfile at state 0, we would see an entry for it, but it would\nshow whatever sha1 the ref had the last time it was packed\n(which could be an arbitrarily long time ago).\n\nThis can be especially dangerous when process A is \"git\nprune\", as it means our set of reachable tips will be\nincomplete, and we may erroneously prune objects reachable\nfrom that tip (the same thing can happen if \"repack -ad\" is\nused, as it simply drops unreachable objects that are\npacked).\n\nThis patch solves it by loading all of the loose refs for\nour traversal into our in-memory cache, and then refreshing\nthe packed-refs cache. Because a pack-refs writer will\nalways put the new packed-refs file into place before\nstarting the prune, we know that any loose refs we fail to\nsee will either truly be missing, or will have already been\nput in the packed-refs file by the time we refresh.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 39 +++++++++++++++++++++++++++++++++++----\n 1 file changed, 35 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex d53a172..350054c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -750,6 +750,21 @@ static int do_for_each_entry_in_dirs(struct ref_dir *dir1,\n }\n \n /*\n+ * Load all of the refs from the dir into our in-memory cache. The hard work\n+ * of loading loose refs is done by get_ref_dir(), so we just need to recurse\n+ * through all of the sub-directories. We do not even need to care about\n+ * sorting, as traversal order does not matter to us.\n+ */\n+static void prime_ref_dir(struct ref_dir *dir)\n+{\n+\tint i;\n+\tfor (i = 0; i < dir->nr; i++) {\n+\t\tstruct ref_entry *entry = dir->entries[i];\n+\t\tif (entry->flag & REF_DIR)\n+\t\t\tprime_ref_dir(get_ref_dir(entry));\n+\t}\n+}\n+/*\n  * Return true iff refname1 and refname2 conflict with each other.\n  * Two reference names conflict if one of them exactly matches the\n  * leading components of the other; e.g., \"foo/bar\" conflicts with\n@@ -1600,15 +1615,31 @@ void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)\n static int do_for_each_entry(struct ref_cache *refs, const char *base,\n \t\t\t     each_ref_entry_fn fn, void *cb_data)\n {\n-\tstruct packed_ref_cache *packed_ref_cache = get_packed_ref_cache(refs);\n-\tstruct ref_dir *packed_dir = get_packed_ref_dir(packed_ref_cache);\n-\tstruct ref_dir *loose_dir = get_loose_refs(refs);\n+\tstruct packed_ref_cache *packed_ref_cache;\n+\tstruct ref_dir *loose_dir;\n+\tstruct ref_dir *packed_dir;\n \tint retval = 0;\n \n+\t/*\n+\t * We must make sure that all loose refs are read before accessing the\n+\t * packed-refs file; this avoids a race condition in which loose refs\n+\t * are migrated to the packed-refs file by a simultaneous process, but\n+\t * our in-memory view is from before the migration. get_packed_ref_cache()\n+\t * takes care of making sure our view is up to date with what is on\n+\t * disk.\n+\t */\n+\tloose_dir = get_loose_refs(refs);\n+\tif (base && *base) {\n+\t\tloose_dir = find_containing_dir(loose_dir, base, 0);\n+\t}\n+\tif (loose_dir)\n+\t\tprime_ref_dir(loose_dir);\n+\n+\tpacked_ref_cache = get_packed_ref_cache(refs);\n \tacquire_packed_ref_cache(packed_ref_cache);\n+\tpacked_dir = get_packed_ref_dir(packed_ref_cache);\n \tif (base && *base) {\n \t\tpacked_dir = find_containing_dir(packed_dir, base, 0);\n-\t\tloose_dir = find_containing_dir(loose_dir, base, 0);\n \t}\n \n \tif (packed_dir && loose_dir) {\n-- \n1.8.3.1\n"},{"id":"221351","messageId":"1371628293-28824-13-git-send-email-mhagger@alum.mit.edu","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"[PATCH v2 12/12] refs: do not invalidate the packed-refs cache unnecessarily","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-19T07:51:33Z","receivedAt":"2013-06-19T07:51:33Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Now that we keep track of the packed-refs file metadata, we can detect\nwhen the packed-refs file has been modified since we last read it, and\nwe do so automatically every time that get_packed_ref_cache() is\ncalled.  So there is no need to invalidate the cache automatically\nwhen lock_packed_refs() is called; usually the old copy will still be\nvalid.\n\nSigned-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n---\n refs.c | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 350054c..4b989b0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2133,11 +2133,14 @@ int lock_packed_refs(struct lock_file *lock, int flags)\n {\n \tstruct packed_ref_cache *packed_ref_cache;\n \n-\t/* Discard the old cache because it might be invalid: */\n-\tclear_packed_ref_cache(&ref_cache);\n \tif (hold_lock_file_for_update(lock, git_path(\"packed-refs\"), flags) < 0)\n \t\treturn -1;\n-\t/* Read the current packed-refs while holding the lock: */\n+\t/*\n+\t * Get the current packed-refs while holding the lock.  If the\n+\t * packed-refs file has been modified since we last read it,\n+\t * this will automatically invalidate the cache and re-read\n+\t * the packed-refs file.\n+\t */\n \tpacked_ref_cache = get_packed_ref_cache(&ref_cache);\n \tpacked_ref_cache->lock = lock;\n \t/* Increment the reference count to prevent it from being freed: */\n-- \n1.8.3.1\n"},{"id":"221397","messageId":"20130619185645.GB23647@sigill.intra.peff.net","threadId":"34217","inReplyTo":"1371628293-28824-1-git-send-email-mhagger@alum.mit.edu","subject":"Re: [PATCH v2 00/12] Fix some reference-related races","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-19T18:56:45Z","receivedAt":"2013-06-19T18:56:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 19, 2013 at 09:51:21AM +0200, Michael Haggerty wrote:\n\n> Re-roll of mh/ref-races.  Thanks to Peff, Junio, and Ramsay for\n> reviewing v1.\n\nThanks. I just read through them again. Everything looks good to me.\n\nPatches 10 and 11 are missing my signoff, but obviously:\n\n  Signed-off-by: Jeff King <peff@peff.net>\n\n> The last patch is still optional--it avoids a little bit of work when\n> rewriting the packed-refs file, but relies on the stat-based freshness\n> check not giving a false negative.\n\nI don't have a real problem with it, but given the cygwin confusions\nthat Ramsay mentioned, maybe it is better to hold back on it for now? It\nsounds like the cygwin problems go the other way (false positives\ninstead of false negatives).\n\n-Peff\n"},{"id":"221400","messageId":"7vfvwdzz6k.fsf@alter.siamese.dyndns.org","threadId":"34217","inReplyTo":"1371628293-28824-5-git-send-email-mhagger@alum.mit.edu","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-06-19T19:18:59Z","receivedAt":"2013-06-19T19:18:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty <mhagger@alum.mit.edu> writes:\n\n> Handle simple transactions for the packed-refs file at the\n> packed_ref_cache level via new functions lock_packed_refs(),\n> commit_packed_refs(), and rollback_packed_refs().\n>\n> Only allow the packed ref cache to be modified (via add_packed_ref())\n> while the packed refs file is locked.\n>\n> Change clone to add the new references within a transaction.\n>\n> Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n> ---\n>  builtin/clone.c |  7 ++++-\n>  refs.c          | 83 ++++++++++++++++++++++++++++++++++++++++++++++-----------\n>  refs.h          | 27 +++++++++++++++++--\n>  3 files changed, 98 insertions(+), 19 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 66bff57..b0c000a 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -489,17 +489,22 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n>  \treturn local_refs;\n>  }\n>  \n> +static struct lock_file packed_refs_lock;\n> +\n>  static void write_remote_refs(const struct ref *local_refs)\n>  {\n>  \tconst struct ref *r;\n>  \n> +\tlock_packed_refs(&packed_refs_lock, LOCK_DIE_ON_ERROR);\n> +\n>  \tfor (r = local_refs; r; r = r->next) {\n>  \t\tif (!r->peer_ref)\n>  \t\t\tcontinue;\n>  \t\tadd_packed_ref(r->peer_ref->name, r->old_sha1);\n>  \t}\n>  \n> -\tpack_refs(PACK_REFS_ALL);\n> +\tif (commit_packed_refs())\n> +\t\tdie_errno(\"unable to overwrite old ref-pack file\");\n>  }\n\nThe calling convention used here looks somewhat strange.  You allow\ncallers to specify which lock-file structure is used when locking,\nbut when you are done, commit_packed_refs() does not take any\nparameter.\n\nlock_packed_refs() make the singleton in-core packed-ref-cache be\naware of which lock it is under, so commit_packed_refs() does not\nneed to be told (the singleton already knows what lockfile is in\neffect), so I am not saying the code is broken, though.\n\nDoes the caller need to even have an access to this lock_file\ninstance?\n\n>  static void write_followtags(const struct ref *refs, const char *msg)\n> diff --git a/refs.c b/refs.c\n> index 373d95b..ad73251 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -808,6 +808,13 @@ static int is_refname_available(const char *refname, const char *oldrefname,\n>  \n>  struct packed_ref_cache {\n>  \tstruct ref_entry *root;\n> +\n> +\t/*\n> +\t * Iff the packed-refs file associated with this instance is\n> +\t * currently locked for writing, this points at the associated\n> +\t * lock (which is owned by somebody else).\n> +\t */\n> +\tstruct lock_file *lock;\n>  };\n>  \n>  /*\n> @@ -829,6 +836,8 @@ static struct ref_cache {\n>  static void clear_packed_ref_cache(struct ref_cache *refs)\n>  {\n>  \tif (refs->packed) {\n> +\t\tif (refs->packed->lock)\n> +\t\t\tdie(\"internal error: packed-ref cache cleared while locked\");\n>  \t\tfree_ref_entry(refs->packed->root);\n>  \t\tfree(refs->packed);\n>  \t\trefs->packed = NULL;\n> @@ -1038,7 +1047,12 @@ static struct ref_dir *get_packed_refs(struct ref_cache *refs)\n>  \n>  void add_packed_ref(const char *refname, const unsigned char *sha1)\n>  {\n> -\tadd_ref(get_packed_refs(&ref_cache),\n> +\tstruct packed_ref_cache *packed_ref_cache =\n> +\t\tget_packed_ref_cache(&ref_cache);\n> +\n> +\tif (!packed_ref_cache->lock)\n> +\t\tdie(\"internal error: packed refs not locked\");\n> +\tadd_ref(get_packed_ref_dir(packed_ref_cache),\n>  \t\tcreate_ref_entry(refname, sha1, REF_ISPACKED, 1));\n>  }\n>  \n> @@ -2035,6 +2049,52 @@ static int write_packed_entry_fn(struct ref_entry *entry, void *cb_data)\n>  \treturn 0;\n>  }\n>  \n> +int lock_packed_refs(struct lock_file *lock, int flags)\n> +{\n> +\tstruct packed_ref_cache *packed_ref_cache;\n> +\n> +\t/* Discard the old cache because it might be invalid: */\n> +\tclear_packed_ref_cache(&ref_cache);\n> +\tif (hold_lock_file_for_update(lock, git_path(\"packed-refs\"), flags) < 0)\n> +\t\treturn -1;\n> +\t/* Read the current packed-refs while holding the lock: */\n> +\tpacked_ref_cache = get_packed_ref_cache(&ref_cache);\n> +\tpacked_ref_cache->lock = lock;\n> +\treturn 0;\n> +}\n> +\n> +int commit_packed_refs(void)\n> +{\n> +\tstruct packed_ref_cache *packed_ref_cache =\n> +\t\tget_packed_ref_cache(&ref_cache);\n> +\tint error = 0;\n> +\n> +\tif (!packed_ref_cache->lock)\n> +\t\tdie(\"internal error: packed-refs not locked\");\n> +\twrite_or_die(packed_ref_cache->lock->fd,\n> +\t\t     PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n> +\n> +\tdo_for_each_entry_in_dir(get_packed_ref_dir(packed_ref_cache),\n> +\t\t\t\t 0, write_packed_entry_fn,\n> +\t\t\t\t &packed_ref_cache->lock->fd);\n> +\tif (commit_lock_file(packed_ref_cache->lock))\n> +\t\terror = -1;\n> +\tpacked_ref_cache->lock = NULL;\n> +\treturn error;\n> +}\n> +\n> +void rollback_packed_refs(void)\n> +{\n> +\tstruct packed_ref_cache *packed_ref_cache =\n> +\t\tget_packed_ref_cache(&ref_cache);\n> +\n> +\tif (!packed_ref_cache->lock)\n> +\t\tdie(\"internal error: packed-refs not locked\");\n> +\trollback_lock_file(packed_ref_cache->lock);\n> +\tpacked_ref_cache->lock = NULL;\n> +\tclear_packed_ref_cache(&ref_cache);\n> +}\n> +\n>  struct ref_to_prune {\n>  \tstruct ref_to_prune *next;\n>  \tunsigned char sha1[20];\n> @@ -2153,23 +2213,19 @@ static struct lock_file packlock;\n>  int pack_refs(unsigned int flags)\n>  {\n>  \tstruct pack_refs_cb_data cbdata;\n> -\tint fd;\n>  \n>  \tmemset(&cbdata, 0, sizeof(cbdata));\n>  \tcbdata.flags = flags;\n>  \n> -\tfd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"),\n> -\t\t\t\t       LOCK_DIE_ON_ERROR);\n> +\tlock_packed_refs(&packlock, LOCK_DIE_ON_ERROR);\n>  \tcbdata.packed_refs = get_packed_refs(&ref_cache);\n>  \n>  \tdo_for_each_entry_in_dir(get_loose_refs(&ref_cache), 0,\n>  \t\t\t\t pack_if_possible_fn, &cbdata);\n>  \n> -\twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n> -\tdo_for_each_entry_in_dir(cbdata.packed_refs, 0, write_packed_entry_fn, &fd);\n> -\n> -\tif (commit_lock_file(&packlock) < 0)\n> +\tif (commit_packed_refs())\n>  \t\tdie_errno(\"unable to overwrite old ref-pack file\");\n> +\n>  \tprune_refs(cbdata.ref_to_prune);\n>  \treturn 0;\n>  }\n> @@ -2233,7 +2289,6 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n>  \n>  static int repack_without_ref(const char *refname)\n>  {\n> -\tint fd;\n>  \tstruct ref_dir *packed;\n>  \tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n>  \tstruct string_list_item *ref_to_delete;\n> @@ -2241,12 +2296,10 @@ static int repack_without_ref(const char *refname)\n>  \tif (!get_packed_ref(refname))\n>  \t\treturn 0; /* refname does not exist in packed refs */\n>  \n> -\tfd = hold_lock_file_for_update(&packlock, git_path(\"packed-refs\"), 0);\n> -\tif (fd < 0) {\n> +\tif (lock_packed_refs(&packlock, 0)) {\n>  \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n>  \t\treturn error(\"cannot delete '%s' from packed refs\", refname);\n>  \t}\n> -\tclear_packed_ref_cache(&ref_cache);\n>  \tpacked = get_packed_refs(&ref_cache);\n>  \n>  \t/* Remove refname from the cache: */\n> @@ -2255,7 +2308,7 @@ static int repack_without_ref(const char *refname)\n>  \t\t * The packed entry disappeared while we were\n>  \t\t * acquiring the lock.\n>  \t\t */\n> -\t\trollback_lock_file(&packlock);\n> +\t\trollback_packed_refs();\n>  \t\treturn 0;\n>  \t}\n>  \n> @@ -2267,9 +2320,7 @@ static int repack_without_ref(const char *refname)\n>  \t}\n>  \n>  \t/* Write what remains: */\n> -\twrite_or_die(fd, PACKED_REFS_HEADER, strlen(PACKED_REFS_HEADER));\n> -\tdo_for_each_entry_in_dir(packed, 0, write_packed_entry_fn, &fd);\n> -\treturn commit_lock_file(&packlock);\n> +\treturn commit_packed_refs();\n>  }\n>  \n>  int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n> diff --git a/refs.h b/refs.h\n> index 246bf60..b275124 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -77,12 +77,35 @@ extern int for_each_rawref(each_ref_fn, void *);\n>  extern void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname);\n>  \n>  /*\n> - * Add a reference to the in-memory packed reference cache.  To actually\n> - * write the reference to the packed-refs file, call pack_refs().\n> + * Lock the packed-refs file for writing using the specified lock_file\n> + * instance.  Flags is passed to hold_lock_file_for_update().  Return\n> + * 0 on success.\n> + */\n> +extern int lock_packed_refs(struct lock_file *lock, int flags);\n> +\n> +/*\n> + * Add a reference to the in-memory packed reference cache.  This may\n> + * only be called while the packed-refs file is locked (see\n> + * lock_packed_refs()).  To actually write the packed-refs file, call\n> + * commit_packed_refs().\n>   */\n>  extern void add_packed_ref(const char *refname, const unsigned char *sha1);\n>  \n>  /*\n> + * Write the current version of the packed refs cache from memory to\n> + * disk.  The packed-refs file must already be locked for writing (see\n> + * lock_packed_refs()).  Return zero on success.\n> + */\n> +extern int commit_packed_refs(void);\n> +\n> +/*\n> + * Rollback the lockfile for the packed-refs file, and discard the\n> + * in-memory packed reference cache.  (The packed-refs file will be\n> + * read anew if it is needed again after this function is called.)\n> + */\n> +extern void rollback_packed_refs(void);\n> +\n> +/*\n>   * Flags for controlling behaviour of pack_refs()\n>   * PACK_REFS_PRUNE: Prune loose refs after packing\n>   * PACK_REFS_ALL:   Pack _all_ refs, not just tags and already packed refs\n"},{"id":"221430","messageId":"51C2B41F.2050708@alum.mit.edu","threadId":"34217","inReplyTo":"7vfvwdzz6k.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-20T07:49:51Z","receivedAt":"2013-06-20T07:49:51Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 06/19/2013 09:18 PM, Junio C Hamano wrote:\n> Michael Haggerty <mhagger@alum.mit.edu> writes:\n> \n>> Handle simple transactions for the packed-refs file at the\n>> packed_ref_cache level via new functions lock_packed_refs(),\n>> commit_packed_refs(), and rollback_packed_refs().\n>>\n>> Only allow the packed ref cache to be modified (via add_packed_ref())\n>> while the packed refs file is locked.\n>>\n>> Change clone to add the new references within a transaction.\n>>\n>> Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>\n>> ---\n>>  builtin/clone.c |  7 ++++-\n>>  refs.c          | 83 ++++++++++++++++++++++++++++++++++++++++++++++-----------\n>>  refs.h          | 27 +++++++++++++++++--\n>>  3 files changed, 98 insertions(+), 19 deletions(-)\n>>\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index 66bff57..b0c000a 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -489,17 +489,22 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n>>  \treturn local_refs;\n>>  }\n>>  \n>> +static struct lock_file packed_refs_lock;\n>> +\n>>  static void write_remote_refs(const struct ref *local_refs)\n>>  {\n>>  \tconst struct ref *r;\n>>  \n>> +\tlock_packed_refs(&packed_refs_lock, LOCK_DIE_ON_ERROR);\n>> +\n>>  \tfor (r = local_refs; r; r = r->next) {\n>>  \t\tif (!r->peer_ref)\n>>  \t\t\tcontinue;\n>>  \t\tadd_packed_ref(r->peer_ref->name, r->old_sha1);\n>>  \t}\n>>  \n>> -\tpack_refs(PACK_REFS_ALL);\n>> +\tif (commit_packed_refs())\n>> +\t\tdie_errno(\"unable to overwrite old ref-pack file\");\n>>  }\n> \n> The calling convention used here looks somewhat strange.  You allow\n> callers to specify which lock-file structure is used when locking,\n> but when you are done, commit_packed_refs() does not take any\n> parameter.\n> \n> lock_packed_refs() make the singleton in-core packed-ref-cache be\n> aware of which lock it is under, so commit_packed_refs() does not\n> need to be told (the singleton already knows what lockfile is in\n> effect), so I am not saying the code is broken, though.\n> \n> Does the caller need to even have an access to this lock_file\n> instance?\n\nNo it doesn't.  My reasoning was as follows:\n\nlock_file instances can never be freed, because they are added to a\nlinked list in the atexit handler but never removed.  Therefore they\nhave to be allocated statically (or allocated dynamically then leaked).\n\n[I just noticed that lock_ref_sha1_basic() leaks a struct lock_file\nevery time that it is called.]\n\nI wanted to leave the way open to allow other packed refs caches to be\nlocked.  But since all packed refs caches are allocated dynamically, the\nlock_file instance cannot be part of struct packed_ref_cache.  So I\nthought that the easiest approach is to rely on the caller, who\npresumably can know that only a finite number of locks are in use at\nonce, to supply a usable lock_file instance.\n\nBut currently only the main packed ref cache can be locked, so it would\nbe possible for lock_packed_refs() to use the static packlock instance\nfor locking.  I will change it to do so.\n\nIf/when we add support for locking other packed ref caches, we can\nchange the API to use a caller-supplied lock_file.  Or even better,\nimplement a way to remove a lock_file instance from the atexit list;\nthen lock_packed_refs() could use dynamically-allocated lock_file\ninstances without having to leak them.\n\nv3 to come.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"221451","messageId":"51C2C6DD.90107@alum.mit.edu","threadId":"34217","inReplyTo":"20130619185645.GB23647@sigill.intra.peff.net","subject":"Re: [PATCH v2 00/12] Fix some reference-related races","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-20T09:09:49Z","receivedAt":"2013-06-20T09:09:49Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 06/19/2013 08:56 PM, Jeff King wrote:\n> On Wed, Jun 19, 2013 at 09:51:21AM +0200, Michael Haggerty wrote:\n> \n>> Re-roll of mh/ref-races.  Thanks to Peff, Junio, and Ramsay for\n>> reviewing v1.\n> \n> Thanks. I just read through them again. Everything looks good to me.\n> \n> Patches 10 and 11 are missing my signoff, but obviously:\n> \n>   Signed-off-by: Jeff King <peff@peff.net>\n> \n>> The last patch is still optional--it avoids a little bit of work when\n>> rewriting the packed-refs file, but relies on the stat-based freshness\n>> check not giving a false negative.\n> \n> I don't have a real problem with it, but given the cygwin confusions\n> that Ramsay mentioned, maybe it is better to hold back on it for now? It\n> sounds like the cygwin problems go the other way (false positives\n> instead of false negatives).\n\nI just realized that there is another long-standing problem in\nloose/packed refs handling:\n\nSuppose there is a loose reference for which an old version is still\npresent in the packed-refs file.  If the reference is deleted:\n\n1. delete_ref() gets a lock on the loose reference file\n\n2. delete_ref() deletes the loose reference file while retaining the lock\n\n3. delete_ref() calls repack_without_ref()\n\n4. repack_without_ref() tries to acquire a lock on the packed-refs file\nto delete the packed version of the reference.\n\nIf the packed-refs file is already locked by another process (and there\nis no reason why that cannot be, and there is only one attempt to\nacquire the lock), then repack_without_ref() emits an error and returns\nwith an error code.  delete_ref() passes the error along, but doesn't\nrestore the loose ref.\n\nThe net result is that an old version of the ref (namely the one in the\nloose refs file) has been revived.  That version might even point at an\nobject that has already been garbage collected.\n\nThis problem is similar to race conditions that have been discussed\nearlier, but this problem has nothing to do with the freshness/staleness\nof the packed-refs file WRT to the loose reference; it only depends on\nthe packed-refs file being locked by another process at an inopportune time.\n\nI think this problem would also be fixed by the locking scheme that I\nproposed earlier [1]: to acquire and hold the packed-refs lock across\nthe modification of *both* files, and to rewrite the packed-refs file\n*before* deleting the loose-refs file (because rewriting the packed-refs\nfile without the to-be-deleted reference is a logical NOP).\n\nMichael\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/212131\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"221465","messageId":"20130620115221.GA773@sigill.intra.peff.net","threadId":"34217","inReplyTo":"51C2C6DD.90107@alum.mit.edu","subject":"Re: [PATCH v2 00/12] Fix some reference-related races","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-20T11:52:22Z","receivedAt":"2013-06-20T11:52:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 20, 2013 at 11:09:49AM +0200, Michael Haggerty wrote:\n\n> If the packed-refs file is already locked by another process (and there\n> is no reason why that cannot be, and there is only one attempt to\n> acquire the lock), then repack_without_ref() emits an error and returns\n> with an error code.  delete_ref() passes the error along, but doesn't\n> restore the loose ref.\n>\n> [...]\n>\n> I think this problem would also be fixed by the locking scheme that I\n> proposed earlier [1]: to acquire and hold the packed-refs lock across\n> the modification of *both* files, and to rewrite the packed-refs file\n> *before* deleting the loose-refs file (because rewriting the packed-refs\n> file without the to-be-deleted reference is a logical NOP).\n\nYeah, I agree. You could also \"roll back\" the loose deletion, but I'd\nrather just try to do it atomically.\n\nI don't think this increases lock contention, since delete_ref would\nneed to lock the packed-refs file anyway. However, there is the related\nchange that we should probably lock the packed-refs file before checking\n\"is this ref in the packed-refs file?\" in repack_without_ref. Which does\nincrease lock contention, but is more correct.\n\nWe should also consider deadlock issues. If the order is always \"acquire\npacked-refs lock, then acquire loose locks\", we are fine. If this does\nloose-then-packed, we are also fine with the current code, as\ngit-pack-refs does not prune the loose refs while under the packed-refs\nlock. But I seem to recall discussion of pruning them under the\npacked-refs lock, which would deadlock if repack_without_ref does\nloose-then-packed.\n\nBut I guess we do not actually block on locks, but rather just die (and\nrelease our locks), so deadlock is not an option for us.\n\n-Peff\n"},{"id":"221466","messageId":"20130620115508.GB773@sigill.intra.peff.net","threadId":"34217","inReplyTo":"51C2B41F.2050708@alum.mit.edu","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-20T11:55:09Z","receivedAt":"2013-06-20T11:55:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 20, 2013 at 09:49:51AM +0200, Michael Haggerty wrote:\n\n> [I just noticed that lock_ref_sha1_basic() leaks a struct lock_file\n> every time that it is called.]\n\nI noticed that recently, too. I have a patch series about 90% complete\nthat abstracts the tempfile handling (the ultimate goal of which is to\noptionally clean up tmp_* files in the objects/ directory). It refactors\nthe lockfile cleanup, and it would not be too hard to have a committed\nor rolled-back lockfile actually remove itself from the \"to clean at\nexit\" list.\n\nWhich would make it perfectly safe to have a lockfile as an automatic\nvariable as long as you commit or rollback before leaving the function.\n\n-Peff\n"},{"id":"221495","messageId":"7vd2rgwvvd.fsf@alter.siamese.dyndns.org","threadId":"34217","inReplyTo":"51C2B41F.2050708@alum.mit.edu","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-06-20T17:11:02Z","receivedAt":"2013-06-20T17:11:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty <mhagger@alum.mit.edu> writes:\n\n> But currently only the main packed ref cache can be locked, so it would\n> be possible for lock_packed_refs() to use the static packlock instance\n> for locking.\n\nPerhaps I am missing something from the previous discussions, but I\nam having trouble understanding the \"main packed ref cache\" part of\nthe above.  \"main\" as opposed to...?  Is it envisioned that later\nsomebody can lock one subpart while another can lock a different and\nnon-overlapping subpart, to make changes independently, and somehow\ntheir non-overlapping changes will be consolidated into a single\nconsistent result?\n"},{"id":"221497","messageId":"51C342B7.5050002@alum.mit.edu","threadId":"34217","inReplyTo":"7vd2rgwvvd.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-20T17:58:15Z","receivedAt":"2013-06-20T17:58:15Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 06/20/2013 07:11 PM, Junio C Hamano wrote:\n> Michael Haggerty <mhagger@alum.mit.edu> writes:\n> \n>> But currently only the main packed ref cache can be locked, so it would\n>> be possible for lock_packed_refs() to use the static packlock instance\n>> for locking.\n> \n> Perhaps I am missing something from the previous discussions, but I\n> am having trouble understanding the \"main packed ref cache\" part of\n> the above.  \"main\" as opposed to...?\n\n\"main\" as opposed to \"submodule\".\n\n> Is it envisioned that later\n> somebody can lock one subpart while another can lock a different and\n> non-overlapping subpart, to make changes independently, and somehow\n> their non-overlapping changes will be consolidated into a single\n> consistent result?\n\nNo, the scenario would be that a git process wants to change a reference\nin a submodule directly, as opposed to starting another git process\nwithin the submodule, as I believe is done now.  Maybe it's too\nfar-fetched even to consider...\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"221499","messageId":"51C343FF.6030002@alum.mit.edu","threadId":"34217","inReplyTo":"20130620115508.GB773@sigill.intra.peff.net","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-06-20T18:03:43Z","receivedAt":"2013-06-20T18:03:43Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 06/20/2013 01:55 PM, Jeff King wrote:\n> On Thu, Jun 20, 2013 at 09:49:51AM +0200, Michael Haggerty wrote:\n> \n>> [I just noticed that lock_ref_sha1_basic() leaks a struct lock_file\n>> every time that it is called.]\n> \n> I noticed that recently, too. I have a patch series about 90% complete\n> that abstracts the tempfile handling (the ultimate goal of which is to\n> optionally clean up tmp_* files in the objects/ directory). It refactors\n> the lockfile cleanup, and it would not be too hard to have a committed\n> or rolled-back lockfile actually remove itself from the \"to clean at\n> exit\" list.\n> \n> Which would make it perfectly safe to have a lockfile as an automatic\n> variable as long as you commit or rollback before leaving the function.\n\nCool, then I won't work on that.  You might also have to make the\nlockfile list into a doubly-linked-list to avoid having to do a linear\nscan to find the entry to delete, unless the total number of entries is\nknown to remain small.\n\nPlease CC me on the patch series when it is done.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"221501","messageId":"7vip18vdcb.fsf@alter.siamese.dyndns.org","threadId":"34217","inReplyTo":"51C342B7.5050002@alum.mit.edu","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-06-20T18:36:36Z","receivedAt":"2013-06-20T18:36:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty <mhagger@alum.mit.edu> writes:\n\n> On 06/20/2013 07:11 PM, Junio C Hamano wrote:\n>\n>> Perhaps I am missing something from the previous discussions, but I\n>> am having trouble understanding the \"main packed ref cache\" part of\n>> the above.  \"main\" as opposed to...?\n>\n> \"main\" as opposed to \"submodule\".\n\nI see.\n\n> No, the scenario would be that a git process wants to change a reference\n> in a submodule directly, as opposed to starting another git process\n> within the submodule, as I believe is done now.  Maybe it's too\n> far-fetched even to consider...\n\nPerhaps.  But the \"singleton lock because we only handle main packed\nref cache\" does not prevent us from doing so later, so I think v3 is\nOK.\n\nThanks.\n"},{"id":"221508","messageId":"20130620195526.GA31364@sigill.intra.peff.net","threadId":"34217","inReplyTo":"51C343FF.6030002@alum.mit.edu","subject":"Re: [PATCH v2 04/12] refs: implement simple transactions for the packed-refs file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-20T19:55:26Z","receivedAt":"2013-06-20T19:55:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 20, 2013 at 08:03:43PM +0200, Michael Haggerty wrote:\n\n> > I noticed that recently, too. I have a patch series about 90% complete\n> > that abstracts the tempfile handling (the ultimate goal of which is to\n> > optionally clean up tmp_* files in the objects/ directory). It refactors\n> > the lockfile cleanup, and it would not be too hard to have a committed\n> > or rolled-back lockfile actually remove itself from the \"to clean at\n> > exit\" list.\n> > \n> > Which would make it perfectly safe to have a lockfile as an automatic\n> > variable as long as you commit or rollback before leaving the function.\n> \n> Cool, then I won't work on that.  You might also have to make the\n> lockfile list into a doubly-linked-list to avoid having to do a linear\n> scan to find the entry to delete, unless the total number of entries is\n> known to remain small.\n\nYes, I noticed that potential issue, but I don't think it is worth\nworrying about. We typically only take one lock at a time, or a handful\nof tempfiles (e.g., one object at a time, or two files for diff).\n\nAnd once it's abstracted out, it would be easy to handle later.\n\nThe part I am a little stuck on is plugging it into\npack-objects/index-pack. Their output handling is a little convoluted\nbecause they may be writing to stdout, to a tempfile, to a named file,\nor even appending to an existing file in the case of index-pack\n--fix-thin. I don't think it's unmanageable, but I need to spend some\nmore time on the refactoring.\n\n> Please CC me on the patch series when it is done.\n\nWill do.\n\n-Peff\n"}]}