{"thread":{"id":"36669","subject":"[PATCH v8 01/44] refs.c: constify the sha arguments for ref_transaction_create|delete|update","startedAt":"2014-05-15T17:29:15Z","lastAt":"2014-05-28T22:23:53Z","messageCount":139,"participants":["Ronnie Sahlberg","Jonathan Nieder","Junio C Hamano","Jeff King","Michael Haggerty"],"isPatch":true,"patchVersion":8,"patchTotal":44},"messages":[{"id":"241741","messageId":"1400174999-26786-1-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":null,"subject":"[PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:15Z","receivedAt":"2014-05-15T17:29:15Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"This patch series can also be found at\nhttps://github.com/rsahlberg/git/tree/ref-transactions\n\n\nThis patch series is based on next and expands on the transaction API. It\nconverts all ref updates, inside refs.c as well as external, to use the\ntransaction API for updates. This makes most of the ref updates to become\natomic when there are failures locking or writing to a ref.\n\nThis version completes the work to convert all ref updates to use transactions.\nNow that all updates are through transactions I will start working on\ncleaning up the reading of refs and to create an api for managing reflogs but\nall that will go in a different patch series.\n\nVersion 8:\n - Updates after review by JN\n - Improve commit messages\n - Add a patch that adds an err argument to repack_without_refs\n - Add a patch that adds an err argument to delete_loose_ref\n - Better document that a _update/_delete/_create failure means the whole\n transaction has failed and needs rollback.\n\nVersion 7:\n - Updated commit messages per JNs review comments.\n - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not\n exposed through refs.h\n\nVersion 6:\n - Convert all updates in refs.c to use transactions too.\n\nVersion 5:\n - Reword commit messages for having _create/_delete/_update returning\n success/failure. There are no conditions yet that return an error from\n these failures but there will be in the future. So we still check the\n return from these functions in the callers in preparation for this.\n - Don't leak memory by just passing a strbuf_detach() pointer to functions.\n Use .buf and explicitely strbuf_release the data afterwards.\n - Remove the function update_ref_lock.\n - Remove the function update_ref_write.\n - Track transaction status and die(BUG:) if we call _create/_delete/_update/\n _commit for a transaction that is not OPEN.\n\nVersion 4:\n - Rename patch series from \"Use ref transactions from most callers\" to\n \"Use ref transactions for all ref updates\".\n - Convert all external ref writes to use transactions and make write_ref_sha1\n and lock_ref_sha1 static functions.\n - Change the ref commit and free handling so we no longer pass pointer to\n pointer to _commit. _commit no longer frees the transaction. The caller\n MUST call _free itself.\n - Change _commit to take a strbuf pointer instead of a char* for error\n reporting back to the caller.\n - Re-add the walker patch after fixing it.\n\nVersion 3:\n - Remove the walker patch for now. Walker needs more complex solution\n so defer it until the basics are done.\n - Remove the onerr argument to ref_transaction_commit(). All callers\n that need to die() on error now have to do this explicitely.\n - Pass an error string from ref_transaction_commit() back to the callers\n so that they can craft a nice error message upon failures.\n - Make ref_transaction_rollback() accept NULL as argument.\n - Change ref_transaction_commit() to take a pointer to pointer argument for\n the transaction and have it clear the callers pointer to NULL when\n invoked. This allows for much nicer handling of transaction rollback on\n failure.\nVersion 2:\n - Add a patch to ref_transaction_commit to make it honor onerr even if the\n error triggered in ref_Transaction_commit itself rather than in a call\n to other functions (that already honor onerr).\n - Add a patch to make the update_ref() helper function use transactions\n internally.\n - Change ref_transaction_update to die() instead of error() if we pass\n if a NULL old_sha1 but have have_old == true.\n - Change ref_transaction_create to die() instead of error() if new_sha1\n is false but we pass it a null_sha1.\n - Change ref_transaction_delete die() instead of error() if we pass\n if a NULL old_sha1 but have have_old == true.\n - Change several places to do if(!transaction || ref_transaction_update()\n || ref_Transaction_commit()) die(generic-message) instead of checking each\n step separately and having a different message for each failure.\n Most users are likely not interested in what step of the transaction\n failed and only whether it failed or not.\n - Change commit.c to only pass a pointer to ref_transaction_update\n iff current_head is non-NULL.\n The previous patch used to compute a garbage pointer for\n current_head->object.sha1 and relied on the fact that ref_transaction_update\n would not try to dereference this pointer if !!current_head was 0.\n - Updated commit message for the walker_fetch change to try to justify why\n the change in locking semantics should not be harmful.\n\n\n\nRonnie Sahlberg (44):\n refs.c: constify the sha arguments for\n ref_transaction_create|delete|update\n refs.c: allow passing NULL to ref_transaction_free\n refs.c: add a strbuf argument to ref_transaction_commit for error\n logging\n refs.c: add an err argument to repack_without_refs\n refs.c: make ref_update_reject_duplicates take a strbuf argument for\n errors\n refs.c: add an err argument ro delete_loose_ref\n refs.c: make update_ref_write update a strbuf on failure\n update-ref.c: log transaction error from the update_ref\n refs.c: remove the onerr argument to ref_transaction_commit\n refs.c: change ref_transaction_update() to do error checking and\n return status\n refs.c: change ref_transaction_create to do error checking and return\n status\n refs.c: ref_transaction_delete to check for error and return status\n tag.c: use ref transactions when doing updates\n replace.c: use the ref transaction functions for updates\n commit.c: use ref transactions for updates\n sequencer.c: use ref transactions for all ref updates\n fast-import.c: change update_branch to use ref transactions\n branch.c: use ref transaction for all ref updates\n refs.c: change update_ref to use a transaction\n refs.c: free the transaction before returning when number of updates\n is 0\n refs.c: ref_transaction_commit should not free the transaction\n fetch.c: clear errno before calling functions that might set it\n fetch.c: change s_update_ref to use a ref transaction\n fetch.c: use a single ref transaction for all ref updates\n receive-pack.c: use a reference transaction for updating the refs\n fast-import.c: use a ref transaction when dumping tags\n walker.c: use ref transaction for ref updates\n refs.c: make write_ref_sha1 static\n refs.c: make lock_ref_sha1 static\n refs.c: add transaction.status and track OPEN/CLOSED/ERROR\n refs.c: remove the update_ref_lock function\n refs.c: remove the update_ref_write function\n refs.c: remove lock_ref_sha1\n refs.c: make prune_ref use a transaction to delete the ref\n refs.c: make delete_ref use a transaction\n refs.c: pass the ref log message to _create/delete/update instead of\n _commit\n refs.c: pass NULL as *flags to read_ref_full\n refs.c: pack all refs before we start to rename a ref\n refs.c: move the check for valid refname to lock_ref_sha1_basic\n refs.c: call lock_ref_sha1_basic directly from commit\n refs.c: add a new flag for transaction delete for refs we know are\n packed only\n refs.c: pass a skip list to name_conflict_fn\n refs.c: make rename_ref use a transaction\n refs.c: remove forward declaration of write_ref_sha1\n\n branch.c | 31 ++--\n builtin/commit.c | 24 ++-\n builtin/fetch.c | 29 ++--\n builtin/receive-pack.c | 20 +--\n builtin/replace.c | 15 +-\n builtin/tag.c | 15 +-\n builtin/update-ref.c | 32 ++--\n fast-import.c | 39 +++--\n refs.c | 443 ++++++++++++++++++++++++++++---------------------\n refs.h | 54 +++---\n sequencer.c | 24 ++-\n t/t3200-branch.sh | 2 +-\n walker.c | 51 +++---\n 13 files changed, 449 insertions(+), 330 deletions(-)\n\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241704","messageId":"1400174999-26786-2-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 01/44] refs.c: constify the sha arguments for ref_transaction_create|delete|update","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:16Z","receivedAt":"2014-05-15T17:29:16Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"ref_transaction_create|delete|update has no need to modify the sha1\narguments passed to it so it should use const unsigned char* instead\nof unsigned char*.\n\nSome functions, such as fast_forward_to(), already have its old/new\nsha1 arguments as consts. This function will at some point need to\nuse ref_transaction_update() in which case this change is required.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 7 ++++---\n refs.h | 7 ++++---\n 2 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 6898263..2114748 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3338,7 +3338,8 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \n void ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *new_sha1, unsigned char *old_sha1,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const unsigned char *old_sha1,\n \t\t\t int flags, int have_old)\n {\n \tstruct ref_update *update = add_update(transaction, refname);\n@@ -3352,7 +3353,7 @@ void ref_transaction_update(struct ref_transaction *transaction,\n \n void ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *new_sha1,\n+\t\t\t const unsigned char *new_sha1,\n \t\t\t int flags)\n {\n \tstruct ref_update *update = add_update(transaction, refname);\n@@ -3366,7 +3367,7 @@ void ref_transaction_create(struct ref_transaction *transaction,\n \n void ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *old_sha1,\n+\t\t\t const unsigned char *old_sha1,\n \t\t\t int flags, int have_old)\n {\n \tstruct ref_update *update = add_update(transaction, refname);\ndiff --git a/refs.h b/refs.h\nindex 09ff483..6d97700 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -245,7 +245,8 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n */\n void ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *new_sha1, unsigned char *old_sha1,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const unsigned char *old_sha1,\n \t\t\t int flags, int have_old);\n \n /*\n@@ -256,7 +257,7 @@ void ref_transaction_update(struct ref_transaction *transaction,\n */\n void ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *new_sha1,\n+\t\t\t const unsigned char *new_sha1,\n \t\t\t int flags);\n \n /*\n@@ -266,7 +267,7 @@ void ref_transaction_create(struct ref_transaction *transaction,\n */\n void ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n-\t\t\t unsigned char *old_sha1,\n+\t\t\t const unsigned char *old_sha1,\n \t\t\t int flags, int have_old);\n \n /*\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241748","messageId":"1400174999-26786-3-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:17Z","receivedAt":"2014-05-15T17:29:17Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Allow ref_transaction_free to be called with NULL and as a result allow\nref_transaction_rollback to be called for a NULL transaction.\n\nThis allows us to write code that will\n if ( (!transaction ||\n ref_transaction_update(...)) ||\n (ref_transaction_commit(...) && !(transaction = NULL)) {\n ref_transaction_rollback(transaction);\n ...\n }\n\nIn this case transaction is reset to NULL IFF ref_transaction_commit() was\ninvoked and thus the rollback becomes ref_transaction_rollback(NULL) which\nis safe. IF the conditional triggered prior to ref_transaction_commit()\nthen transaction is untouched and then ref_transaction_rollback(transaction)\nwill rollback the failed transaction.\n\nReviewed-by: Jonathan Nieder \nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 2114748..88d73c8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3312,6 +3312,9 @@ static void ref_transaction_free(struct ref_transaction *transaction)\n {\n \tint i;\n \n+\tif (!transaction)\n+\t\treturn;\n+\n \tfor (i = 0; i < transaction->nr; i++)\n \t\tfree(transaction->updates[i]);\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241737","messageId":"1400174999-26786-4-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 03/44] refs.c: add a strbuf argument to ref_transaction_commit for error logging","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:18Z","receivedAt":"2014-05-15T17:29:18Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Add a strbuf argument to _commit so that we can pass an error string back to\nthe caller. So that we can do error logging from the caller instead of from\n_commit.\n\nLonger term plan is to first convert all callers to use onerr==QUIET_ON_ERR\nand craft any log messages from the callers themselves and finally remove the\nonerr argument completely.\n\nReviewed-by: Jonathan Nieder \nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 2 +-\n refs.c | 6 +++++-\n refs.h | 5 ++++-\n 3 files changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 405267f..aaa06aa 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -367,7 +367,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \t\tif (end_null)\n \t\t\tline_termination = '\\0';\n \t\tupdate_refs_stdin();\n-\t\tret = ref_transaction_commit(transaction, msg,\n+\t\tret = ref_transaction_commit(transaction, msg, NULL,\n \t\t\t\t\t UPDATE_REFS_DIE_ON_ERR);\n \t\treturn ret;\n \t}\ndiff --git a/refs.c b/refs.c\nindex 88d73c8..1a7f9d9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3423,7 +3423,8 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n }\n \n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, enum action_on_err onerr)\n+\t\t\t const char *msg, struct strbuf *err,\n+\t\t\t enum action_on_err onerr)\n {\n \tint ret = 0, delnum = 0, i;\n \tconst char **delnames;\n@@ -3452,6 +3453,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t update->flags,\n \t\t\t\t\t &update->type, onerr);\n \t\tif (!update->lock) {\n+\t\t\tif (err)\n+\t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n+\t\t\t\t\t update->refname);\n \t\t\tret = 1;\n \t\t\tgoto cleanup;\n \t\t}\ndiff --git a/refs.h b/refs.h\nindex 6d97700..25ae110 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -274,9 +274,12 @@ void ref_transaction_delete(struct ref_transaction *transaction,\n * Commit all of the changes that have been queued in transaction, as\n * atomically as possible. Return a nonzero value if there is a\n * problem. The ref_transaction is freed by this function.\n+ * If err is non-NULL we will add an error string to it to explain why\n+ * the transaction failed. The string does not end in newline.\n */\n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, enum action_on_err onerr);\n+\t\t\t const char *msg, struct strbuf *err,\n+\t\t\t enum action_on_err onerr);\n \n /** Lock a ref and then write its file */\n int update_ref(const char *action, const char *refname,\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241705","messageId":"1400174999-26786-5-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 04/44] refs.c: add an err argument to repack_without_refs","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:19Z","receivedAt":"2014-05-15T17:29:19Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Update repack_without_refs to take an err argument and update it if there\nis a failure. Pass the err variable from ref_transaction_commit to this\nfunction so that callers can print a meaningful error message if _commit\nfails due to a problem in repack_without_refs.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1a7f9d9..c2b720e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n \treturn 0;\n }\n \n-static int repack_without_refs(const char **refnames, int n)\n+static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n {\n \tstruct ref_dir *packed;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *ref_to_delete;\n-\tint i, removed = 0;\n+\tint i, ret, removed = 0;\n \n \t/* Look for a packed ref */\n \tfor (i = 0; i < n; i++)\n@@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)\n \n \tif (lock_packed_refs(0)) {\n \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n+\t\tif (err)\n+\t\t\tstrbuf_addf(err, \"cannot delete '%s' from packed refs\",\n+\t\t\t\t refnames[i]);\n \t\treturn error(\"cannot delete '%s' from packed refs\", refnames[i]);\n \t}\n \tpacked = get_packed_refs(&ref_cache);\n@@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)\n \t}\n \n \t/* Write what remains */\n-\treturn commit_packed_refs();\n+\tret = commit_packed_refs();\n+\tif (ret && err)\n+\t\tstrbuf_addf(err, \"unable to overwrite old ref-pack file\");\n+\treturn ret;\n }\n \n static int repack_without_ref(const char *refname)\n {\n-\treturn repack_without_refs(&refname, 1);\n+\treturn repack_without_refs(&refname, 1, NULL);\n }\n \n static int delete_ref_loose(struct ref_lock *lock, int flag)\n@@ -3486,7 +3492,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t}\n \t}\n \n-\tret |= repack_without_refs(delnames, delnum);\n+\tret |= repack_without_refs(delnames, delnum, err);\n \tfor (i = 0; i < delnum; i++)\n \t\tunlink_or_warn(git_path(\"logs/%s\", delnames[i]));\n \tclear_loose_ref_cache(&ref_cache);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241744","messageId":"1400174999-26786-6-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 05/44] refs.c: make ref_update_reject_duplicates take a strbuf argument for errors","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:20Z","receivedAt":"2014-05-15T17:29:20Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Make ref_update_reject_duplicates return any error that occurs through a\nnew strbuf argument. This means that when a transaction commit fails in\nthis function we will now be able to pass a helpful error message back to the\ncaller.\n\nReviewed-by: Jonathan Nieder \nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex c2b720e..072e3e9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3408,6 +3408,7 @@ static int ref_update_compare(const void *r1, const void *r2)\n }\n \n static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n+\t\t\t\t\tstruct strbuf *err,\n \t\t\t\t\tenum action_on_err onerr)\n {\n \tint i;\n@@ -3415,6 +3416,9 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n \t\tif (!strcmp(updates[i - 1]->refname, updates[i]->refname)) {\n \t\t\tconst char *str =\n \t\t\t\t\"Multiple updates for ref '%s' not allowed.\";\n+\t\t\tif (err)\n+\t\t\t\tstrbuf_addf(err, str, updates[i]->refname);\n+\n \t\t\tswitch (onerr) {\n \t\t\tcase UPDATE_REFS_MSG_ON_ERR:\n \t\t\t\terror(str, updates[i]->refname); break;\n@@ -3445,7 +3449,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \t/* Copy, sort, and reject duplicate refs */\n \tqsort(updates, n, sizeof(*updates), ref_update_compare);\n-\tret = ref_update_reject_duplicates(updates, n, onerr);\n+\tret = ref_update_reject_duplicates(updates, n, err, onerr);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241746","messageId":"1400174999-26786-7-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:21Z","receivedAt":"2014-05-15T17:29:21Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Add an err argument to delete_loose_ref so that we can pass a descriptive\nerror string back to the caller. Pass the err argument from transaction\ncommit to this function so that transaction users will have a nice error\nstring if the transaction failed due to delete_loose_ref failing.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 072e3e9..a150520 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)\n \treturn repack_without_refs(&refname, 1, NULL);\n }\n \n-static int delete_ref_loose(struct ref_lock *lock, int flag)\n+static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n {\n \tif (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {\n \t\t/* loose */\n-\t\tint err, i = strlen(lock->lk->filename) - 5; /* .lock */\n+\t\tint res, i = strlen(lock->lk->filename) - 5; /* .lock */\n \n \t\tlock->lk->filename[i] = 0;\n-\t\terr = unlink_or_warn(lock->lk->filename);\n+\t\tres = unlink_or_warn(lock->lk->filename);\n \t\tlock->lk->filename[i] = '.';\n-\t\tif (err && errno != ENOENT)\n+\t\tif (res && errno != ENOENT) {\n+\t\t\tif (err)\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t \"failed to delete loose ref '%s'\",\n+\t\t\t\t\t lock->lk->filename);\n \t\t\treturn 1;\n+\t\t}\n \t}\n \treturn 0;\n }\n@@ -2507,7 +2512,7 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n \tlock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);\n \tif (!lock)\n \t\treturn 1;\n-\tret |= delete_ref_loose(lock, flag);\n+\tret |= delete_ref_loose(lock, flag, NULL);\n \n \t/* removing the loose one could have resurrected an earlier\n \t * packed one. Also, if it was not loose we need to repack\n@@ -3492,7 +3497,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \t\tif (update->lock) {\n \t\t\tdelnames[delnum++] = update->lock->ref_name;\n-\t\t\tret |= delete_ref_loose(update->lock, update->type);\n+\t\t\tret |= delete_ref_loose(update->lock, update->type,\n+\t\t\t\t\t\terr);\n \t\t}\n \t}\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241742","messageId":"1400174999-26786-8-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 07/44] refs.c: make update_ref_write update a strbuf on failure","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:22Z","receivedAt":"2014-05-15T17:29:22Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change update_ref_write to also update an error strbuf on failure.\nThis makes the error available to ref_transaction_commit callers if the\ntransaction failed due to update_ref_sha1/write_ref_sha1 failures.\n\nReviewed-by: Jonathan Nieder \nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex a150520..18dff1b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3273,10 +3273,13 @@ static struct ref_lock *update_ref_lock(const char *refname,\n \n static int update_ref_write(const char *action, const char *refname,\n \t\t\t const unsigned char *sha1, struct ref_lock *lock,\n-\t\t\t enum action_on_err onerr)\n+\t\t\t struct strbuf *err, enum action_on_err onerr)\n {\n \tif (write_ref_sha1(lock, sha1, action) < 0) {\n \t\tconst char *str = \"Cannot update the ref '%s'.\";\n+\t\tif (err)\n+\t\t\tstrbuf_addf(err, str, refname);\n+\n \t\tswitch (onerr) {\n \t\tcase UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;\n \t\tcase UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;\n@@ -3402,7 +3405,7 @@ int update_ref(const char *action, const char *refname,\n \tlock = update_ref_lock(refname, oldval, flags, NULL, onerr);\n \tif (!lock)\n \t\treturn 1;\n-\treturn update_ref_write(action, refname, sha1, lock, onerr);\n+\treturn update_ref_write(action, refname, sha1, lock, NULL, onerr);\n }\n \n static int ref_update_compare(const void *r1, const void *r2)\n@@ -3484,7 +3487,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\tret = update_ref_write(msg,\n \t\t\t\t\t update->refname,\n \t\t\t\t\t update->new_sha1,\n-\t\t\t\t\t update->lock, onerr);\n+\t\t\t\t\t update->lock, err, onerr);\n \t\t\tupdate->lock = NULL; /* freed by update_ref_write */\n \t\t\tif (ret)\n \t\t\t\tgoto cleanup;\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241747","messageId":"1400174999-26786-9-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 08/44] update-ref.c: log transaction error from the update_ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:23Z","receivedAt":"2014-05-15T17:29:23Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Call ref_transaction_commit with QUIET_ON_ERR and use the strbuf that is\nreturned to print a log message if/after the transaction fails.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex aaa06aa..207e24d 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -342,6 +342,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \tconst char *refname, *oldval, *msg = NULL;\n \tunsigned char sha1[20], oldsha1[20];\n \tint delete = 0, no_deref = 0, read_stdin = 0, end_null = 0, flags = 0;\n+\tstruct strbuf err = STRBUF_INIT;\n \tstruct option options[] = {\n \t\tOPT_STRING( 'm', NULL, &msg, N_(\"reason\"), N_(\"reason of the update\")),\n \t\tOPT_BOOL('d', NULL, &delete, N_(\"delete the reference\")),\n@@ -359,17 +360,16 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \t\tdie(\"Refusing to perform update with empty message.\");\n \n \tif (read_stdin) {\n-\t\tint ret;\n \t\ttransaction = ref_transaction_begin();\n-\n \t\tif (delete || no_deref || argc > 0)\n \t\t\tusage_with_options(git_update_ref_usage, options);\n \t\tif (end_null)\n \t\t\tline_termination = '\\0';\n \t\tupdate_refs_stdin();\n-\t\tret = ref_transaction_commit(transaction, msg, NULL,\n-\t\t\t\t\t UPDATE_REFS_DIE_ON_ERR);\n-\t\treturn ret;\n+\t\tif (ref_transaction_commit(transaction, msg, &err,\n+\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR))\n+\t\t\tdie(\"%s\", err.buf);\n+\t\treturn 0;\n \t}\n \n \tif (end_null)\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241745","messageId":"1400174999-26786-10-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 09/44] refs.c: remove the onerr argument to ref_transaction_commit","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:24Z","receivedAt":"2014-05-15T17:29:24Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Since all callers now use QUIET_ON_ERR we no longer need to provide an onerr\nargument any more. Remove the onerr argument from the ref_transaction_commit\nsignature.\n\nReviewed-by: Jonathan Nieder \nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 3 +--\n refs.c | 22 +++++++---------------\n refs.h | 3 +--\n 3 files changed, 9 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 207e24d..2bef2a0 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -366,8 +366,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \t\tif (end_null)\n \t\t\tline_termination = '\\0';\n \t\tupdate_refs_stdin();\n-\t\tif (ref_transaction_commit(transaction, msg, &err,\n-\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR))\n+\t\tif (ref_transaction_commit(transaction, msg, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\treturn 0;\n \t}\ndiff --git a/refs.c b/refs.c\nindex 18dff1b..bc21060 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3416,8 +3416,7 @@ static int ref_update_compare(const void *r1, const void *r2)\n }\n \n static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n-\t\t\t\t\tstruct strbuf *err,\n-\t\t\t\t\tenum action_on_err onerr)\n+\t\t\t\t\tstruct strbuf *err)\n {\n \tint i;\n \tfor (i = 1; i < n; i++)\n@@ -3427,22 +3426,13 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n \t\t\tif (err)\n \t\t\t\tstrbuf_addf(err, str, updates[i]->refname);\n \n-\t\t\tswitch (onerr) {\n-\t\t\tcase UPDATE_REFS_MSG_ON_ERR:\n-\t\t\t\terror(str, updates[i]->refname); break;\n-\t\t\tcase UPDATE_REFS_DIE_ON_ERR:\n-\t\t\t\tdie(str, updates[i]->refname); break;\n-\t\t\tcase UPDATE_REFS_QUIET_ON_ERR:\n-\t\t\t\tbreak;\n-\t\t\t}\n \t\t\treturn 1;\n \t\t}\n \treturn 0;\n }\n \n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, struct strbuf *err,\n-\t\t\t enum action_on_err onerr)\n+\t\t\t const char *msg, struct strbuf *err)\n {\n \tint ret = 0, delnum = 0, i;\n \tconst char **delnames;\n@@ -3457,7 +3447,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \t/* Copy, sort, and reject duplicate refs */\n \tqsort(updates, n, sizeof(*updates), ref_update_compare);\n-\tret = ref_update_reject_duplicates(updates, n, err, onerr);\n+\tret = ref_update_reject_duplicates(updates, n, err);\n \tif (ret)\n \t\tgoto cleanup;\n \n@@ -3469,7 +3459,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t (update->have_old ?\n \t\t\t\t\t\tupdate->old_sha1 : NULL),\n \t\t\t\t\t update->flags,\n-\t\t\t\t\t &update->type, onerr);\n+\t\t\t\t\t &update->type,\n+\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR);\n \t\tif (!update->lock) {\n \t\t\tif (err)\n \t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n@@ -3487,7 +3478,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\tret = update_ref_write(msg,\n \t\t\t\t\t update->refname,\n \t\t\t\t\t update->new_sha1,\n-\t\t\t\t\t update->lock, err, onerr);\n+\t\t\t\t\t update->lock, err,\n+\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR);\n \t\t\tupdate->lock = NULL; /* freed by update_ref_write */\n \t\t\tif (ret)\n \t\t\t\tgoto cleanup;\ndiff --git a/refs.h b/refs.h\nindex 25ae110..555ee59 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -278,8 +278,7 @@ void ref_transaction_delete(struct ref_transaction *transaction,\n * the transaction failed. The string does not end in newline.\n */\n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, struct strbuf *err,\n-\t\t\t enum action_on_err onerr);\n+\t\t\t const char *msg, struct strbuf *err);\n \n /** Lock a ref and then write its file */\n int update_ref(const char *action, const char *refname,\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241738","messageId":"1400174999-26786-11-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:25Z","receivedAt":"2014-05-15T17:29:25Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Update ref_transaction_update() do some basic error checking and return\nnon-zero on error. Update all callers to check ref_transaction_update() for\nerror. There are currently no conditions in _update that will return error but\nthere will be in the future.\n\nAlso check for BUGs during update and die(BUG:...) if we are calling\n_update with have_old but the old_sha1 pointer is NULL.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 10 ++++++----\n refs.c | 9 +++++++--\n refs.h | 13 ++++++++-----\n 3 files changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 2bef2a0..59c4d6b 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)\n \tif (*next != line_termination)\n \t\tdie(\"update %s: extra input: %s\", refname, next);\n \n-\tref_transaction_update(transaction, refname, new_sha1, old_sha1,\n-\t\t\t update_flags, have_old);\n+\tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n+\t\t\t\t update_flags, have_old))\n+\t\tdie(\"update %s: failed\", refname);\n \n \tupdate_flags = 0;\n \tfree(refname);\n@@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)\n \tif (*next != line_termination)\n \t\tdie(\"verify %s: extra input: %s\", refname, next);\n \n-\tref_transaction_update(transaction, refname, new_sha1, old_sha1,\n-\t\t\t update_flags, have_old);\n+\tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n+\t\t\t\t update_flags, have_old))\n+\t\tdie(\"failed transaction update for %s\", refname);\n \n \tupdate_flags = 0;\n \tfree(refname);\ndiff --git a/refs.c b/refs.c\nindex bc21060..e8c2345 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3353,19 +3353,24 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \treturn update;\n }\n \n-void ref_transaction_update(struct ref_transaction *transaction,\n+int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *new_sha1,\n \t\t\t const unsigned char *old_sha1,\n \t\t\t int flags, int have_old)\n {\n-\tstruct ref_update *update = add_update(transaction, refname);\n+\tstruct ref_update *update;\n+\n+\tif (have_old && !old_sha1)\n+\t\tdie(\"BUG: have_old is true but old_sha1 is NULL\");\n \n+\tupdate = add_update(transaction, refname);\n \thashcpy(update->new_sha1, new_sha1);\n \tupdate->flags = flags;\n \tupdate->have_old = have_old;\n \tif (have_old)\n \t\thashcpy(update->old_sha1, old_sha1);\n+\treturn 0;\n }\n \n void ref_transaction_create(struct ref_transaction *transaction,\ndiff --git a/refs.h b/refs.h\nindex 555ee59..fd8d731 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n * be deleted. If have_old is true, then old_sha1 holds the value\n * that the reference should have had before the update, or zeros if\n * it must not have existed beforehand.\n+ * Function returns 0 on success and non-zero on failure. A failure to update\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n-void ref_transaction_update(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *new_sha1,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old);\n+int ref_transaction_update(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t int flags, int have_old);\n \n /*\n * Add a reference creation to transaction. new_sha1 is the value\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241708","messageId":"1400174999-26786-12-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:26Z","receivedAt":"2014-05-15T17:29:26Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Do basic error checking in ref_transaction_create() and make it return\nnon-zero on error. Update all callers to check the result of\nref_transaction_create(). There are currently no conditions in _create that\nwill return error but there will be in the future.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 4 +++-\n refs.c | 17 +++++++++++------\n refs.h | 11 +++++++----\n 3 files changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 59c4d6b..0924502 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -225,7 +225,9 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)\n \tif (*next != line_termination)\n \t\tdie(\"create %s: extra input: %s\", refname, next);\n \n-\tref_transaction_create(transaction, refname, new_sha1, update_flags);\n+\tif (ref_transaction_create(transaction, refname, new_sha1,\n+\t\t\t\t update_flags))\n+\t\tdie(\"cannot create ref '%s'\", refname);\n \n \tupdate_flags = 0;\n \tfree(refname);\ndiff --git a/refs.c b/refs.c\nindex e8c2345..f38d3d6 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3373,18 +3373,23 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \treturn 0;\n }\n \n-void ref_transaction_create(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *new_sha1,\n-\t\t\t int flags)\n+int ref_transaction_create(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t int flags)\n {\n-\tstruct ref_update *update = add_update(transaction, refname);\n+\tstruct ref_update *update;\n+\n+\tif (!new_sha1 || is_null_sha1(new_sha1))\n+\t\tdie(\"BUG: create ref with null new_sha1\");\n+\n+\tupdate = add_update(transaction, refname);\n \n-\tassert(!is_null_sha1(new_sha1));\n \thashcpy(update->new_sha1, new_sha1);\n \thashclr(update->old_sha1);\n \tupdate->flags = flags;\n \tupdate->have_old = 1;\n+\treturn 0;\n }\n \n void ref_transaction_delete(struct ref_transaction *transaction,\ndiff --git a/refs.h b/refs.h\nindex fd8d731..e3bdd57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -257,11 +257,14 @@ int ref_transaction_update(struct ref_transaction *transaction,\n * that the reference should have after the update; it must not be the\n * null SHA-1. It is verified that the reference does not exist\n * already.\n+ * Function returns 0 on success and non-zero on failure. A failure to create\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n-void ref_transaction_create(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *new_sha1,\n-\t\t\t int flags);\n+int ref_transaction_create(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t int flags);\n \n /*\n * Add a reference deletion to transaction. If have_old is true, then\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241740","messageId":"1400174999-26786-13-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:27Z","receivedAt":"2014-05-15T17:29:27Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change ref_transaction_delete() to do basic error checking and return\nnon-zero of error. Update all callers to check the return for\nref_transaction_delete(). There are currently no conditions in _delete that\nwill return error but there will be in the future.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/update-ref.c | 5 +++--\n refs.c | 15 ++++++++++-----\n refs.h | 11 +++++++----\n 3 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 0924502..2a51df1 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -257,8 +257,9 @@ static const char *parse_cmd_delete(struct strbuf *input, const char *next)\n \tif (*next != line_termination)\n \t\tdie(\"delete %s: extra input: %s\", refname, next);\n \n-\tref_transaction_delete(transaction, refname, old_sha1,\n-\t\t\t update_flags, have_old);\n+\tif (ref_transaction_delete(transaction, refname, old_sha1,\n+\t\t\t\t update_flags, have_old))\n+\t\tdie(\"failed transaction delete for %s\", refname);\n \n \tupdate_flags = 0;\n \tfree(refname);\ndiff --git a/refs.c b/refs.c\nindex f38d3d6..6e5e940 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3392,19 +3392,24 @@ int ref_transaction_create(struct ref_transaction *transaction,\n \treturn 0;\n }\n \n-void ref_transaction_delete(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old)\n+int ref_transaction_delete(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t int flags, int have_old)\n {\n-\tstruct ref_update *update = add_update(transaction, refname);\n+\tstruct ref_update *update;\n \n+\tif (have_old && !old_sha1)\n+\t\tdie(\"BUG: have_old is true but old_sha1 is NULL\");\n+\n+\tupdate = add_update(transaction, refname);\n \tupdate->flags = flags;\n \tupdate->have_old = have_old;\n \tif (have_old) {\n \t\tassert(!is_null_sha1(old_sha1));\n \t\thashcpy(update->old_sha1, old_sha1);\n \t}\n+\treturn 0;\n }\n \n int update_ref(const char *action, const char *refname,\ndiff --git a/refs.h b/refs.h\nindex e3bdd57..cba2f3d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -270,11 +270,14 @@ int ref_transaction_create(struct ref_transaction *transaction,\n * Add a reference deletion to transaction. If have_old is true, then\n * old_sha1 holds the value that the reference should have had before\n * the update (which must not be the null SHA-1).\n+ * Function returns 0 on success and non-zero on failure. A failure to delete\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n-void ref_transaction_delete(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old);\n+int ref_transaction_delete(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t int flags, int have_old);\n \n /*\n * Commit all of the changes that have been queued in transaction, as\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241743","messageId":"1400174999-26786-14-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 13/44] tag.c: use ref transactions when doing updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:28Z","receivedAt":"2014-05-15T17:29:28Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change tag.c to use ref transactions for all ref updates.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/tag.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex c6e8a71..bf2a5c3 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -548,7 +548,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct strbuf ref = STRBUF_INIT;\n \tunsigned char object[20], prev[20];\n \tconst char *object_ref, *tag;\n-\tstruct ref_lock *lock;\n \tstruct create_tag_options opt;\n \tchar *cleanup_arg = NULL;\n \tint annotate = 0, force = 0, lines = -1;\n@@ -556,6 +555,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tconst char *msgfile = NULL, *keyid = NULL;\n \tstruct msg_arg msg = { 0, STRBUF_INIT };\n \tstruct commit_list *with_commit = NULL;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n \tstruct option options[] = {\n \t\tOPT_CMDMODE('l', \"list\", &cmdmode, N_(\"list tag names\"), 'l'),\n \t\t{ OPTION_INTEGER, 'n', NULL, &lines, N_(\"n\"),\n@@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tif (annotate)\n \t\tcreate_tag(object, tag, &buf, &opt, prev, object);\n \n-\tlock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);\n-\tif (!lock)\n-\t\tdie(_(\"%s: cannot lock the ref\"), ref.buf);\n-\tif (write_ref_sha1(lock, object, NULL) < 0)\n-\t\tdie(_(\"%s: cannot update the ref\"), ref.buf);\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_update(transaction, ref.buf, object, prev,\n+\t\t\t\t 0, !is_null_sha1(prev)) ||\n+\t ref_transaction_commit(transaction, NULL, &err))\n+\t\tdie(_(\"%s: cannot update the ref: %s\"), ref.buf, err.buf);\n \tif (force && !is_null_sha1(prev) && hashcmp(prev, object))\n \t\tprintf(_(\"Updated tag '%s' (was %s)\\n\"), tag, find_unique_abbrev(prev, DEFAULT_ABBREV));\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241721","messageId":"1400174999-26786-15-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 14/44] replace.c: use the ref transaction functions for updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:29Z","receivedAt":"2014-05-15T17:29:29Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Update replace.c to use ref transactions for updates.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/replace.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/replace.c b/builtin/replace.c\nindex 3da1bae..ee34d5d 100644\n--- a/builtin/replace.c\n+++ b/builtin/replace.c\n@@ -133,7 +133,8 @@ static int replace_object_sha1(const char *object_ref,\n \tunsigned char prev[20];\n \tenum object_type obj_type, repl_type;\n \tchar ref[PATH_MAX];\n-\tstruct ref_lock *lock;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n \n \tif (snprintf(ref, sizeof(ref),\n \t\t \"refs/replace/%s\",\n@@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,\n \telse if (!force)\n \t\tdie(\"replace ref '%s' already exists\", ref);\n \n-\tlock = lock_any_ref_for_update(ref, prev, 0, NULL);\n-\tif (!lock)\n-\t\tdie(\"%s: cannot lock the ref\", ref);\n-\tif (write_ref_sha1(lock, repl, NULL) < 0)\n-\t\tdie(\"%s: cannot update the ref\", ref);\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_update(transaction, ref, repl, prev,\n+\t\t\t\t 0, !is_null_sha1(prev)) ||\n+\t ref_transaction_commit(transaction, NULL, &err))\n+\t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n \n \treturn 0;\n }\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241735","messageId":"1400174999-26786-16-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 15/44] commit.c: use ref transactions for updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:30Z","receivedAt":"2014-05-15T17:29:30Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change commit.c to use ref transactions for all ref updates.\nMake sure we pass a NULL pointer to ref_transaction_update if have_old\nis false.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/commit.c | 23 ++++++++++-------------\n 1 file changed, 10 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex d28505a..592f019 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1581,11 +1581,12 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tconst char *index_file, *reflog_msg;\n \tchar *nl;\n \tunsigned char sha1[20];\n-\tstruct ref_lock *ref_lock;\n \tstruct commit_list *parents = NULL, **pptr = &parents;\n \tstruct stat statbuf;\n \tstruct commit *current_head = NULL;\n \tstruct commit_extra_header *extra = NULL;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_commit_usage, builtin_commit_options);\n@@ -1707,16 +1708,6 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&author_ident);\n \tfree_commit_extra_headers(extra);\n \n-\tref_lock = lock_any_ref_for_update(\"HEAD\",\n-\t\t\t\t\t !current_head\n-\t\t\t\t\t ? NULL\n-\t\t\t\t\t : current_head->object.sha1,\n-\t\t\t\t\t 0, NULL);\n-\tif (!ref_lock) {\n-\t\trollback_index_files();\n-\t\tdie(_(\"cannot lock HEAD ref\"));\n-\t}\n-\n \tnl = strchr(sb.buf, '\\n');\n \tif (nl)\n \t\tstrbuf_setlen(&sb, nl + 1 - sb.buf);\n@@ -1725,9 +1716,15 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tstrbuf_insert(&sb, 0, reflog_msg, strlen(reflog_msg));\n \tstrbuf_insert(&sb, strlen(reflog_msg), \": \", 2);\n \n-\tif (write_ref_sha1(ref_lock, sha1, sb.buf) < 0) {\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_update(transaction, \"HEAD\", sha1,\n+\t\t\t\t current_head ?\n+\t\t\t\t current_head->object.sha1 : NULL,\n+\t\t\t\t 0, !!current_head) ||\n+\t ref_transaction_commit(transaction, sb.buf, &err)) {\n \t\trollback_index_files();\n-\t\tdie(_(\"cannot update HEAD ref\"));\n+\t\tdie(\"%s\", err.buf);\n \t}\n \n \tunlink(git_path(\"CHERRY_PICK_HEAD\"));\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241713","messageId":"1400174999-26786-17-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 16/44] sequencer.c: use ref transactions for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:31Z","receivedAt":"2014-05-15T17:29:31Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change to use ref transactions for all updates to refs.\n\nSigned-off-by: Ronnie Sahlberg \n---\n sequencer.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 0a80c58..9282a12 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -272,23 +272,31 @@ static int error_dirty_index(struct replay_opts *opts)\n static int fast_forward_to(const unsigned char *to, const unsigned char *from,\n \t\t\tint unborn, struct replay_opts *opts)\n {\n-\tstruct ref_lock *ref_lock;\n+\tstruct ref_transaction *transaction;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret;\n+\tstruct strbuf err = STRBUF_INIT;\n \n \tread_cache();\n \tif (checkout_fast_forward(from, to, 1))\n \t\texit(1); /* the callee should have complained already */\n-\tref_lock = lock_any_ref_for_update(\"HEAD\", unborn ? null_sha1 : from,\n-\t\t\t\t\t 0, NULL);\n-\tif (!ref_lock)\n-\t\treturn error(_(\"Failed to lock HEAD during fast_forward_to\"));\n \n \tstrbuf_addf(&sb, \"%s: fast-forward\", action_name(opts));\n-\tret = write_ref_sha1(ref_lock, to, sb.buf);\n+\n+\ttransaction = ref_transaction_begin();\n+\tif ((!transaction ||\n+\t ref_transaction_update(transaction, \"HEAD\", to, from,\n+\t\t\t\t 0, !unborn)) ||\n+\t (ref_transaction_commit(transaction, sb.buf, &err) &&\n+\t !(transaction = NULL))) {\n+\t\tref_transaction_rollback(transaction);\n+\t\terror(_(\"HEAD: Could not fast-forward: %s\\n\"), err.buf);\n+\t\tstrbuf_release(&sb);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n \n \tstrbuf_release(&sb);\n-\treturn ret;\n+\treturn 0;\n }\n \n static int do_recursive_merge(struct commit *base, struct commit *next,\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241714","messageId":"1400174999-26786-18-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:32Z","receivedAt":"2014-05-15T17:29:32Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change update_branch() to use ref transactions for updates.\n\nSigned-off-by: Ronnie Sahlberg \n---\n fast-import.c | 22 ++++++++++++++--------\n 1 file changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 6707a66..79d219b 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1679,39 +1679,45 @@ found_entry:\n static int update_branch(struct branch *b)\n {\n \tstatic const char *msg = \"fast-import\";\n-\tstruct ref_lock *lock;\n+\tstruct ref_transaction *transaction;\n \tunsigned char old_sha1[20];\n+\tstruct strbuf err = STRBUF_INIT;\n \n \tif (read_ref(b->name, old_sha1))\n \t\thashclr(old_sha1);\n+\n \tif (is_null_sha1(b->sha1)) {\n \t\tif (b->delete)\n \t\t\tdelete_ref(b->name, old_sha1, 0);\n \t\treturn 0;\n \t}\n-\tlock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);\n-\tif (!lock)\n-\t\treturn error(\"Unable to lock %s\", b->name);\n \tif (!force_update && !is_null_sha1(old_sha1)) {\n \t\tstruct commit *old_cmit, *new_cmit;\n \n \t\told_cmit = lookup_commit_reference_gently(old_sha1, 0);\n \t\tnew_cmit = lookup_commit_reference_gently(b->sha1, 0);\n \t\tif (!old_cmit || !new_cmit) {\n-\t\t\tunlock_ref(lock);\n \t\t\treturn error(\"Branch %s is missing commits.\", b->name);\n \t\t}\n \n \t\tif (!in_merge_bases(old_cmit, new_cmit)) {\n-\t\t\tunlock_ref(lock);\n \t\t\twarning(\"Not updating %s\"\n \t\t\t\t\" (new tip %s does not contain %s)\",\n \t\t\t\tb->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));\n \t\t\treturn -1;\n \t\t}\n \t}\n-\tif (write_ref_sha1(lock, b->sha1, msg) < 0)\n-\t\treturn error(\"Unable to update %s\", b->name);\n+\ttransaction = ref_transaction_begin();\n+\tif ((!transaction ||\n+\t ref_transaction_update(transaction, b->name, b->sha1, old_sha1,\n+\t\t\t\t 0, 1)) ||\n+\t (ref_transaction_commit(transaction, msg, &err) &&\n+\t !(transaction = NULL))) {\n+\t\tref_transaction_rollback(transaction);\n+\t\terror(\"Unable to update branch %s: %s\", b->name, err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n \treturn 0;\n }\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241739","messageId":"1400174999-26786-19-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 18/44] branch.c: use ref transaction for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:33Z","receivedAt":"2014-05-15T17:29:33Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change create_branch to use a ref transaction when creating the new branch.\nref_transaction_create will check that the ref does not already exist and fail\notherwise meaning that we no longer need to keep a lock on the ref during the\nsetup_tracking. This simplifies the code since we can now do the transaction\nin one single step.\n\nIf the forcing flag is false then use ref_transaction_create since this will\nfail if the ref already exist. Otherwise use ref_transaction_update.\n\nThis also fixes a race condition in the old code where two concurrent\ncreate_branch could race since the lock_any_ref_for_update/write_ref_sha1\ndid not protect against the ref already existsing. I.e. one thread could end up\noverwriting a branch even if the forcing flag is false.\n\nSigned-off-by: Ronnie Sahlberg \n---\n branch.c | 30 ++++++++++++++++--------------\n 1 file changed, 16 insertions(+), 14 deletions(-)\n\ndiff --git a/branch.c b/branch.c\nindex 660097b..2aa5c82 100644\n--- a/branch.c\n+++ b/branch.c\n@@ -226,7 +226,6 @@ void create_branch(const char *head,\n \t\t int force, int reflog, int clobber_head,\n \t\t int quiet, enum branch_track track)\n {\n-\tstruct ref_lock *lock = NULL;\n \tstruct commit *commit;\n \tunsigned char sha1[20];\n \tchar *real_ref, msg[PATH_MAX + 20];\n@@ -285,15 +284,6 @@ void create_branch(const char *head,\n \t\tdie(_(\"Not a valid branch point: '%s'.\"), start_name);\n \thashcpy(sha1, commit->object.sha1);\n \n-\tif (!dont_change_ref) {\n-\t\tlock = lock_any_ref_for_update(ref.buf, NULL, 0, NULL);\n-\t\tif (!lock)\n-\t\t\tdie_errno(_(\"Failed to lock ref for update\"));\n-\t}\n-\n-\tif (reflog)\n-\t\tlog_all_ref_updates = 1;\n-\n \tif (forcing)\n \t\tsnprintf(msg, sizeof msg, \"branch: Reset to %s\",\n \t\t\t start_name);\n@@ -301,13 +291,25 @@ void create_branch(const char *head,\n \t\tsnprintf(msg, sizeof msg, \"branch: Created from %s\",\n \t\t\t start_name);\n \n+\tif (reflog)\n+\t\tlog_all_ref_updates = 1;\n+\n+\tif (!dont_change_ref) {\n+\t\tstruct ref_transaction *transaction;\n+\t\tstruct strbuf err = STRBUF_INIT;\n+\n+\t\ttransaction = ref_transaction_begin();\n+\t\tif (!transaction ||\n+\t\t\tref_transaction_update(transaction, ref.buf, sha1,\n+\t\t\t\t\t null_sha1, 0, !forcing) ||\n+\t\t\tref_transaction_commit(transaction, msg, &err))\n+\t\t\t\tdie_errno(_(\"%s: failed to write ref: %s\"),\n+\t\t\t\t\t ref.buf, err.buf);\n+\t}\n+\n \tif (real_ref && track)\n \t\tsetup_tracking(ref.buf + 11, real_ref, track, quiet);\n \n-\tif (!dont_change_ref)\n-\t\tif (write_ref_sha1(lock, sha1, msg) < 0)\n-\t\t\tdie_errno(_(\"Failed to write ref\"));\n-\n \tstrbuf_release(&ref);\n \tfree(real_ref);\n }\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241730","messageId":"1400174999-26786-20-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 19/44] refs.c: change update_ref to use a transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:34Z","receivedAt":"2014-05-15T17:29:34Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change the update_ref helper function to use a ref transaction internally.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 25 +++++++++++++++++++++----\n 1 file changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 6e5e940..0476892 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3416,11 +3416,28 @@ int update_ref(const char *action, const char *refname,\n \t const unsigned char *sha1, const unsigned char *oldval,\n \t int flags, enum action_on_err onerr)\n {\n-\tstruct ref_lock *lock;\n-\tlock = update_ref_lock(refname, oldval, flags, NULL, onerr);\n-\tif (!lock)\n+\tstruct ref_transaction *t;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tt = ref_transaction_begin();\n+\tif ((!t ||\n+\t ref_transaction_update(t, refname, sha1, oldval, flags,\n+\t\t\t\t !!oldval)) ||\n+\t (ref_transaction_commit(t, action, &err) && !(t = NULL))) {\n+\t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n+\n+\t\tref_transaction_rollback(t);\n+\t\tswitch (onerr) {\n+\t\tcase UPDATE_REFS_MSG_ON_ERR:\n+\t\t\terror(str, refname, err.buf); break;\n+\t\tcase UPDATE_REFS_DIE_ON_ERR:\n+\t\t\tdie(str, refname, err.buf); break;\n+\t\tcase UPDATE_REFS_QUIET_ON_ERR: break;\n+\t\t}\n+\t\tstrbuf_release(&err);\n \t\treturn 1;\n-\treturn update_ref_write(action, refname, sha1, lock, NULL, onerr);\n+\t}\n+\treturn 0;\n }\n \n static int ref_update_compare(const void *r1, const void *r2)\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241709","messageId":"1400174999-26786-21-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 20/44] refs.c: free the transaction before returning when number of updates is 0","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:35Z","receivedAt":"2014-05-15T17:29:35Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"We have to free the transaction before returning in the early check for\n'return early if number of updates == 0' or else the following code would\ncreate a memory leak with the transaction never being freed :\n t = ref_transaction_begin()\n ref_transaction_commit(t)\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 0476892..4a37f87 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3471,8 +3471,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n \n-\tif (!n)\n+\tif (!n) {\n+\t\tref_transaction_free(transaction);\n \t\treturn 0;\n+\t}\n \n \t/* Allocate work space */\n \tdelnames = xmalloc(sizeof(*delnames) * n);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241712","messageId":"1400174999-26786-22-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:36Z","receivedAt":"2014-05-15T17:29:36Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change ref_transaction_commit so that it does not free the transaction.\nInstead require that a caller will end a transaction by either calling\nref_transaction_rollback or ref_transaction_free.\n\nBy having the transaction object remaining valid after _commit returns allows\nus to write much nicer code and still be able to call ref_transaction_rollback\nsafely. Instead of this horribleness\n\tt = ref_transaction_begin();\n\tif ((!t ||\n\t ref_transaction_update(t, refname, sha1, oldval, flags,\n\t\t\t\t !!oldval)) ||\n\t (ref_transaction_commit(t, action, &err) && !(t = NULL))) {\n\t\tref_transaction_rollback(t);\n\nwe can now just do the much nicer\n\tt = ref_transaction_begin();\n\tif (!t ||\n\t ref_transaction_update(t, refname, sha1, oldval, flags,\n\t\t\t\t !!oldval) ||\n\t ref_transaction_commit(&t, action, &err)) {\n\t\tref_transaction_rollback(t);\n\t\t... die/return ...\n\n\tref_transaction_free(transaction);\n\nSigned-off-by: Ronnie Sahlberg \n---\n branch.c | 1 +\n builtin/commit.c | 1 +\n builtin/replace.c | 1 +\n builtin/tag.c | 1 +\n builtin/update-ref.c | 1 +\n fast-import.c | 8 ++++----\n refs.c | 14 ++++++--------\n refs.h | 14 +++++++++-----\n sequencer.c | 8 ++++----\n 9 files changed, 28 insertions(+), 21 deletions(-)\n\ndiff --git a/branch.c b/branch.c\nindex 2aa5c82..8e58908 100644\n--- a/branch.c\n+++ b/branch.c\n@@ -305,6 +305,7 @@ void create_branch(const char *head,\n \t\t\tref_transaction_commit(transaction, msg, &err))\n \t\t\t\tdie_errno(_(\"%s: failed to write ref: %s\"),\n \t\t\t\t\t ref.buf, err.buf);\n+\t\tref_transaction_free(transaction);\n \t}\n \n \tif (real_ref && track)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 592f019..16fadbb 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1726,6 +1726,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\trollback_index_files();\n \t\tdie(\"%s\", err.buf);\n \t}\n+\tref_transaction_free(transaction);\n \n \tunlink(git_path(\"CHERRY_PICK_HEAD\"));\n \tunlink(git_path(\"REVERT_HEAD\"));\ndiff --git a/builtin/replace.c b/builtin/replace.c\nindex ee34d5d..91354a7 100644\n--- a/builtin/replace.c\n+++ b/builtin/replace.c\n@@ -164,6 +164,7 @@ static int replace_object_sha1(const char *object_ref,\n \t ref_transaction_commit(transaction, NULL, &err))\n \t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n \n+\tref_transaction_free(transaction);\n \treturn 0;\n }\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex bf2a5c3..fbd2989 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -708,6 +708,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \t\t\t\t 0, !is_null_sha1(prev)) ||\n \t ref_transaction_commit(transaction, NULL, &err))\n \t\tdie(_(\"%s: cannot update the ref: %s\"), ref.buf, err.buf);\n+\tref_transaction_free(transaction);\n \tif (force && !is_null_sha1(prev) && hashcmp(prev, object))\n \t\tprintf(_(\"Updated tag '%s' (was %s)\\n\"), tag, find_unique_abbrev(prev, DEFAULT_ABBREV));\n \ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 2a51df1..b5f4731 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -373,6 +373,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \t\tupdate_refs_stdin();\n \t\tif (ref_transaction_commit(transaction, msg, &err))\n \t\t\tdie(\"%s\", err.buf);\n+\t\tref_transaction_free(transaction);\n \t\treturn 0;\n \t}\n \ndiff --git a/fast-import.c b/fast-import.c\nindex 79d219b..3e356da 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1708,16 +1708,16 @@ static int update_branch(struct branch *b)\n \t\t}\n \t}\n \ttransaction = ref_transaction_begin();\n-\tif ((!transaction ||\n+\tif (!transaction ||\n \t ref_transaction_update(transaction, b->name, b->sha1, old_sha1,\n-\t\t\t\t 0, 1)) ||\n-\t (ref_transaction_commit(transaction, msg, &err) &&\n-\t !(transaction = NULL))) {\n+\t\t\t\t 0, 1) ||\n+\t ref_transaction_commit(transaction, msg, &err)) {\n \t\tref_transaction_rollback(transaction);\n \t\terror(\"Unable to update branch %s: %s\", b->name, err.buf);\n \t\tstrbuf_release(&err);\n \t\treturn -1;\n \t}\n+\tref_transaction_free(transaction);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 4a37f87..82a8d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3322,7 +3322,7 @@ struct ref_transaction *ref_transaction_begin(void)\n \treturn xcalloc(1, sizeof(struct ref_transaction));\n }\n \n-static void ref_transaction_free(struct ref_transaction *transaction)\n+void ref_transaction_free(struct ref_transaction *transaction)\n {\n \tint i;\n \n@@ -3420,10 +3420,10 @@ int update_ref(const char *action, const char *refname,\n \tstruct strbuf err = STRBUF_INIT;\n \n \tt = ref_transaction_begin();\n-\tif ((!t ||\n+\tif (!t ||\n \t ref_transaction_update(t, refname, sha1, oldval, flags,\n-\t\t\t\t !!oldval)) ||\n-\t (ref_transaction_commit(t, action, &err) && !(t = NULL))) {\n+\t\t\t\t !!oldval) ||\n+\t ref_transaction_commit(t, action, &err)) {\n \t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n \n \t\tref_transaction_rollback(t);\n@@ -3437,6 +3437,7 @@ int update_ref(const char *action, const char *refname,\n \t\tstrbuf_release(&err);\n \t\treturn 1;\n \t}\n+\tref_transaction_free(t);\n \treturn 0;\n }\n \n@@ -3471,10 +3472,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n \n-\tif (!n) {\n-\t\tref_transaction_free(transaction);\n+\tif (!n)\n \t\treturn 0;\n-\t}\n \n \t/* Allocate work space */\n \tdelnames = xmalloc(sizeof(*delnames) * n);\n@@ -3541,7 +3540,6 @@ cleanup:\n \t\tif (updates[i]->lock)\n \t\t\tunlock_ref(updates[i]->lock);\n \tfree(delnames);\n-\tref_transaction_free(transaction);\n \treturn ret;\n }\n \ndiff --git a/refs.h b/refs.h\nindex cba2f3d..b94e1ac 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -216,8 +216,8 @@ enum action_on_err {\n \n /*\n * Begin a reference transaction. The reference transaction must\n- * eventually be commited using ref_transaction_commit() or rolled\n- * back using ref_transaction_rollback().\n+ * eventually be freed by either calling ref_transaction_rollback()\n+ * or ref_transaction_free().\n */\n struct ref_transaction *ref_transaction_begin(void);\n \n@@ -282,13 +282,17 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n /*\n * Commit all of the changes that have been queued in transaction, as\n * atomically as possible. Return a nonzero value if there is a\n- * problem. The ref_transaction is freed by this function.\n- * If err is non-NULL we will add an error string to it to explain why\n- * the transaction failed. The string does not end in newline.\n+ * problem. If err is non-NULL we will add an error string to it to explain\n+ * why the transaction failed. The string does not end in newline.\n */\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t const char *msg, struct strbuf *err);\n \n+/*\n+ * Free an existing transaction.\n+ */\n+void ref_transaction_free(struct ref_transaction *transaction);\n+\n /** Lock a ref and then write its file */\n int update_ref(const char *action, const char *refname,\n \t\tconst unsigned char *sha1, const unsigned char *oldval,\ndiff --git a/sequencer.c b/sequencer.c\nindex 9282a12..0cfdaf0 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -283,17 +283,17 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,\n \tstrbuf_addf(&sb, \"%s: fast-forward\", action_name(opts));\n \n \ttransaction = ref_transaction_begin();\n-\tif ((!transaction ||\n+\tif (!transaction ||\n \t ref_transaction_update(transaction, \"HEAD\", to, from,\n-\t\t\t\t 0, !unborn)) ||\n-\t (ref_transaction_commit(transaction, sb.buf, &err) &&\n-\t !(transaction = NULL))) {\n+\t\t\t\t 0, !unborn) ||\n+\t ref_transaction_commit(transaction, sb.buf, &err)) {\n \t\tref_transaction_rollback(transaction);\n \t\terror(_(\"HEAD: Could not fast-forward: %s\\n\"), err.buf);\n \t\tstrbuf_release(&sb);\n \t\tstrbuf_release(&err);\n \t\treturn -1;\n \t}\n+\tref_transaction_free(transaction);\n \n \tstrbuf_release(&sb);\n \treturn 0;\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241715","messageId":"1400174999-26786-23-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:37Z","receivedAt":"2014-05-15T17:29:37Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"In s_update_ref there are two calls that when they fail we return an error\nbased on the errno value. In particular we want to return a specific error\nif ENOTDIR happened. Both these functions do have failure modes where they\nmay return an error without updating errno, in which case a previous and\nunrelated ENOTDIR may cause us to return the wrong error. Clear errno before\ncalling any functions if we check errno afterwards.\n\nAlso skip initializing a static variable to 0. Statics live in .bss and\nare all automatically initialized to 0.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/fetch.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 55f457c..a93c893 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -44,7 +44,7 @@ static struct transport *gtransport;\n static struct transport *gsecondary;\n static const char *submodule_prefix = \"\";\n static const char *recurse_submodules_default;\n-static int shown_url = 0;\n+static int shown_url;\n \n static int option_parse_recurse_submodules(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -382,6 +382,8 @@ static int s_update_ref(const char *action,\n \tif (!rla)\n \t\trla = default_rla.buf;\n \tsnprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n+\n+\terrno = 0;\n \tlock = lock_any_ref_for_update(ref->name,\n \t\t\t\t check_old ? ref->old_sha1 : NULL,\n \t\t\t\t 0, NULL);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241734","messageId":"1400174999-26786-24-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:38Z","receivedAt":"2014-05-15T17:29:38Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change s_update_ref to use a ref transaction for the ref update.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/fetch.c | 17 +++++++++--------\n 1 file changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex a93c893..b41d7b7 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -375,7 +375,7 @@ static int s_update_ref(const char *action,\n {\n \tchar msg[1024];\n \tchar *rla = getenv(\"GIT_REFLOG_ACTION\");\n-\tstatic struct ref_lock *lock;\n+\tstruct ref_transaction *transaction;\n \n \tif (dry_run)\n \t\treturn 0;\n@@ -384,15 +384,16 @@ static int s_update_ref(const char *action,\n \tsnprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n \n \terrno = 0;\n-\tlock = lock_any_ref_for_update(ref->name,\n-\t\t\t\t check_old ? ref->old_sha1 : NULL,\n-\t\t\t\t 0, NULL);\n-\tif (!lock)\n-\t\treturn errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n-\t\t\t\t\t STORE_REF_ERROR_OTHER;\n-\tif (write_ref_sha1(lock, ref->new_sha1, msg) < 0)\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_update(transaction, ref->name, ref->new_sha1,\n+\t\t\t\t ref->old_sha1, 0, check_old) ||\n+\t ref_transaction_commit(transaction, msg, NULL)) {\n+\t\tref_transaction_rollback(transaction);\n \t\treturn errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n \t\t\t\t\t STORE_REF_ERROR_OTHER;\n+\t}\n+\tref_transaction_free(transaction);\n \treturn 0;\n }\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241711","messageId":"1400174999-26786-25-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:39Z","receivedAt":"2014-05-15T17:29:39Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change store_updated_refs to use a single ref transaction for all refs that\nare updated during the fetch. This makes the fetch more atomic when update\nfailures occur.\n\nSince ref update failures will now no longer occur in the code path for\nupdating a single ref in s_update_ref, we no longer have as detailed error\nmessage logging the exact reference and the ref log action as in the old cod\nInstead since we fail the entire transaction we log a much more generic\nmessage. But since we commit the transaction using MSG_ON_ERR we will log\nan error containing the ref name if either locking of writing the ref would\nso the regression in the log message is minor.\n\nThis will also change the order in which errors are checked for and logged\nwhich may alter which error will be logged if there are multiple errors\noccuring during a fetch.\n\nFor example, assume we have a fetch for two refs that both would fail.\nWhere the first ref would fail with ENOTDIR due to a directory in the ref\npath not existing, and the second ref in the fetch would fail due to\nthe check in update_logical_ref():\n if (current_branch &&\n !strcmp(ref->name, current_branch->name) &&\n !(update_head_ok || is_bare_repository()) &&\n !is_null_sha1(ref->old_sha1)) {\n /*\n * If this is the head, and it's not okay to update\n * the head, and the old value of the head isn't empty...\n */\n\nIn the old code since we would update the refs one ref at a time we would\nfirst fail the ENOTDIR and then fail the second update of HEAD as well.\nBut since the first ref failed with ENOTDIR we would eventually fail the who\nfetch with STORE_REF_ERROR_DF_CONFLICT\n\nIn the new code, since we defer committing the transaction until all refs\nhave been processed, we would now detect that the second ref was bad and\nrollback the transaction before we would even try start writing the update t\ndisk and thus we would not return STORE_REF_ERROR_DF_CONFLICT for this case.\n\nI think this new behaviour is more correct, since if there was a problem\nwe would not even try to commit the transaction but need to highlight this\nchange in how/what errors are reported.\nThis change in what error is returned only occurs if there are multiple\nrefs that fail to update and only some, but not all, of them fail due to\nENOTDIR.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/fetch.c | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex b41d7b7..5dbd0f0 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -45,6 +45,7 @@ static struct transport *gsecondary;\n static const char *submodule_prefix = \"\";\n static const char *recurse_submodules_default;\n static int shown_url;\n+struct ref_transaction *transaction;\n \n static int option_parse_recurse_submodules(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -373,27 +374,13 @@ static int s_update_ref(const char *action,\n \t\t\tstruct ref *ref,\n \t\t\tint check_old)\n {\n-\tchar msg[1024];\n-\tchar *rla = getenv(\"GIT_REFLOG_ACTION\");\n-\tstruct ref_transaction *transaction;\n-\n \tif (dry_run)\n \t\treturn 0;\n-\tif (!rla)\n-\t\trla = default_rla.buf;\n-\tsnprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n \n-\terrno = 0;\n-\ttransaction = ref_transaction_begin();\n-\tif (!transaction ||\n-\t ref_transaction_update(transaction, ref->name, ref->new_sha1,\n-\t\t\t\t ref->old_sha1, 0, check_old) ||\n-\t ref_transaction_commit(transaction, msg, NULL)) {\n-\t\tref_transaction_rollback(transaction);\n-\t\treturn errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n-\t\t\t\t\t STORE_REF_ERROR_OTHER;\n-\t}\n-\tref_transaction_free(transaction);\n+\tif (ref_transaction_update(transaction, ref->name, ref->new_sha1,\n+\t\t\t ref->old_sha1, 0, check_old))\n+\t\treturn STORE_REF_ERROR_OTHER;\n+\n \treturn 0;\n }\n \n@@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\tgoto abort;\n \t}\n \n+\terrno = 0;\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction) {\n+\t\trc = error(_(\"cannot start ref transaction\\n\"));\n+\t\tgoto abort;\n+\t}\n+\n \t/*\n \t * We do a pass for each fetch_head_status type in their enum order, so\n \t * merged entries are written before not-for-merge. That lets readers\n@@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\t\t}\n \t\t}\n \t}\n+\tif (ref_transaction_commit(transaction, \"fetch_ref transaction\", NULL))\n+\t\trc |= errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n+\t\t STORE_REF_ERROR_OTHER;\n+\tref_transaction_free(transaction);\n \n \tif (rc & STORE_REF_ERROR_DF_CONFLICT)\n \t\terror(_(\"some local refs could not be updated; try running\\n\"\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241707","messageId":"1400174999-26786-26-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:40Z","receivedAt":"2014-05-15T17:29:40Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Wrap all the ref updates inside a transaction to make the update atomic.\n\nSigned-off-by: Ronnie Sahlberg \n---\n builtin/receive-pack.c | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex c323081..d580176 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -46,6 +46,8 @@ static void *head_name_to_free;\n static int sent_capabilities;\n static int shallow_update;\n static const char *alt_shallow_file;\n+struct strbuf err = STRBUF_INIT;\n+struct ref_transaction *transaction;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -475,7 +477,6 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n \tconst char *namespaced_name;\n \tunsigned char *old_sha1 = cmd->old_sha1;\n \tunsigned char *new_sha1 = cmd->new_sha1;\n-\tstruct ref_lock *lock;\n \n \t/* only refs/... are allowed */\n \tif (!starts_with(name, \"refs/\") || check_refname_format(name + 5, 0)) {\n@@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n \t\t update_shallow_ref(cmd, si))\n \t\t\treturn \"shallow error\";\n \n-\t\tlock = lock_any_ref_for_update(namespaced_name, old_sha1,\n-\t\t\t\t\t 0, NULL);\n-\t\tif (!lock) {\n-\t\t\trp_error(\"failed to lock %s\", name);\n-\t\t\treturn \"failed to lock\";\n-\t\t}\n-\t\tif (write_ref_sha1(lock, new_sha1, \"push\")) {\n-\t\t\treturn \"failed to write\"; /* error() already called */\n-\t\t}\n+\t\tif (ref_transaction_update(transaction, namespaced_name,\n+\t\t\t\t\t new_sha1, old_sha1, 0, 1))\n+\t\t\treturn \"failed to update\";\n \t\treturn NULL; /* good */\n \t}\n }\n@@ -812,6 +807,7 @@ static void execute_commands(struct command *commands,\n \thead_name = head_name_to_free = resolve_refdup(\"HEAD\", sha1, 0, NULL);\n \n \tchecked_connectivity = 1;\n+\ttransaction = ref_transaction_begin();\n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string)\n \t\t\tcontinue;\n@@ -827,6 +823,10 @@ static void execute_commands(struct command *commands,\n \t\t\tchecked_connectivity = 0;\n \t\t}\n \t}\n+\tif (ref_transaction_commit(transaction, \"push\", &err))\n+\t\terror(\"%s\", err.buf);\n+\tref_transaction_free(transaction);\n+\tstrbuf_release(&err);\n \n \tif (shallow_update && !checked_connectivity)\n \t\terror(\"BUG: run 'git fsck' for safety.\\n\"\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241717","messageId":"1400174999-26786-27-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:41Z","receivedAt":"2014-05-15T17:29:41Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \n---\n fast-import.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 3e356da..5587cf6 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1736,15 +1736,22 @@ static void dump_tags(void)\n {\n \tstatic const char *msg = \"fast-import\";\n \tstruct tag *t;\n-\tstruct ref_lock *lock;\n \tchar ref_name[PATH_MAX];\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction;\n \n+\ttransaction = ref_transaction_begin();\n \tfor (t = first_tag; t; t = t->next_tag) {\n-\t\tsprintf(ref_name, \"tags/%s\", t->name);\n-\t\tlock = lock_ref_sha1(ref_name, NULL);\n-\t\tif (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)\n-\t\t\tfailure |= error(\"Unable to update %s\", ref_name);\n+\t\tsprintf(ref_name, \"refs/tags/%s\", t->name);\n+\n+\t\tif (ref_transaction_update(transaction, ref_name, t->sha1,\n+\t\t\t\t\t NULL, 0, 0))\n+\t\t\tfailure |= error(\"Unable to update %s\", err.buf);\n \t}\n+\tif (failure || ref_transaction_commit(transaction, msg, &err))\n+\t\tfailure |= error(\"Unable to update %s\", err.buf);\n+\tref_transaction_free(transaction);\n+\tstrbuf_release(&err);\n }\n \n static void dump_marks_helper(FILE *f,\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241731","messageId":"1400174999-26786-28-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 27/44] walker.c: use ref transaction for ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:42Z","receivedAt":"2014-05-15T17:29:42Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Switch to using ref transactions in walker_fetch(). As part of the refactoring\nto use ref transactions we also fix a potential memory leak where in the\noriginal code if write_ref_sha1() would fail we would end up returning from\nthe function without free()ing the msg string.\n\nThis changes the locking slightly for walker_fetch. Previously the code would\nlock all refs before writing them but now we do not lock the refs until the\ncommit stage. There is thus a very short window where changes could be done\nlocally during the fetch which would be overwritten when the fetch completes\nand commits its transaction. But this window should be reasonably short.\nEven if this race does trigger, since both the old code and the new code\njust overwrites the refs to the new values without checking or comparing\nthem with the previous value, this is not too dissimilar to a similar scenario\nwhere you first do a ref change locally and then later do a fetch that\noverwrites the local change. With this in mind I do not see the change in\nlocking semantics to be critical.\n\nNote that this function is only called when fetching from a remote HTTP\nrepository onto the local (most of the time single-user) repository which\nlikely means that the type of collissions that the previous locking would\nprotect against and cause the fetch to fail for to be even more rare.\n\nSigned-off-by: Ronnie Sahlberg \n---\n walker.c | 51 ++++++++++++++++++++++++++-------------------------\n 1 file changed, 26 insertions(+), 25 deletions(-)\n\ndiff --git a/walker.c b/walker.c\nindex 1dd86b8..6044ccf 100644\n--- a/walker.c\n+++ b/walker.c\n@@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)\n int walker_fetch(struct walker *walker, int targets, char **target,\n \t\t const char **write_ref, const char *write_ref_log_details)\n {\n-\tstruct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));\n+\tchar ref_name[PATH_MAX];\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction;\n \tunsigned char *sha1 = xmalloc(targets * 20);\n-\tchar *msg;\n-\tint ret;\n+\tchar *msg = NULL;\n \tint i;\n \n \tsave_commit_buffer = 0;\n \n-\tfor (i = 0; i < targets; i++) {\n-\t\tif (!write_ref || !write_ref[i])\n-\t\t\tcontinue;\n-\n-\t\tlock[i] = lock_ref_sha1(write_ref[i], NULL);\n-\t\tif (!lock[i]) {\n-\t\t\terror(\"Can't lock ref %s\", write_ref[i]);\n-\t\t\tgoto unlock_and_fail;\n-\t\t}\n-\t}\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction)\n+\t\treturn -1;\n \n \tif (!walker->get_recover)\n \t\tfor_each_ref(mark_complete, NULL);\n@@ -276,14 +270,14 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n \tfor (i = 0; i < targets; i++) {\n \t\tif (interpret_target(walker, target[i], &sha1[20 * i])) {\n \t\t\terror(\"Could not interpret response from server '%s' as something to pull\", target[i]);\n-\t\t\tgoto unlock_and_fail;\n+\t\t\tgoto rollback_and_fail;\n \t\t}\n \t\tif (process(walker, lookup_unknown_object(&sha1[20 * i])))\n-\t\t\tgoto unlock_and_fail;\n+\t\t\tgoto rollback_and_fail;\n \t}\n \n \tif (loop(walker))\n-\t\tgoto unlock_and_fail;\n+\t\tgoto rollback_and_fail;\n \n \tif (write_ref_log_details) {\n \t\tmsg = xmalloc(strlen(write_ref_log_details) + 12);\n@@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n \tfor (i = 0; i < targets; i++) {\n \t\tif (!write_ref || !write_ref[i])\n \t\t\tcontinue;\n-\t\tret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : \"fetch (unknown)\");\n-\t\tlock[i] = NULL;\n-\t\tif (ret)\n-\t\t\tgoto unlock_and_fail;\n+\t\tsprintf(ref_name, \"refs/%s\", write_ref[i]);\n+\t\tif (ref_transaction_update(transaction, ref_name,\n+\t\t\t\t\t &sha1[20 * i], NULL,\n+\t\t\t\t\t 0, 0))\n+\t\t\tgoto rollback_and_fail;\n+\t}\n+\n+\tif (ref_transaction_commit(transaction, msg ? msg : \"fetch (unknown)\",\n+\t\t\t\t &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tgoto rollback_and_fail;\n \t}\n-\tfree(msg);\n \n+\tfree(msg);\n \treturn 0;\n \n-unlock_and_fail:\n-\tfor (i = 0; i < targets; i++)\n-\t\tif (lock[i])\n-\t\t\tunlock_ref(lock[i]);\n+rollback_and_fail:\n+\tfree(msg);\n+\tstrbuf_release(&err);\n+\tref_transaction_free(transaction);\n \n \treturn -1;\n }\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241706","messageId":"1400174999-26786-29-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 28/44] refs.c: make write_ref_sha1 static","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:43Z","receivedAt":"2014-05-15T17:29:43Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"No external users call write_ref_sha1 any more so lets declare it static.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 4 +++-\n refs.h | 3 ---\n 2 files changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 82a8d4e..e5729ad 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -251,6 +251,8 @@ struct ref_entry {\n };\n \n static void read_loose_refs(const char *dirname, struct ref_dir *dir);\n+static int write_ref_sha1(struct ref_lock *lock,\n+\t\t\t const unsigned char *sha1, const char *logmsg);\n \n static struct ref_dir *get_ref_dir(struct ref_entry *entry)\n {\n@@ -2805,7 +2807,7 @@ static int is_branch(const char *refname)\n \treturn !strcmp(refname, \"HEAD\") || starts_with(refname, \"refs/heads/\");\n }\n \n-int write_ref_sha1(struct ref_lock *lock,\n+static int write_ref_sha1(struct ref_lock *lock,\n \tconst unsigned char *sha1, const char *logmsg)\n {\n \tstatic char term = '\\n';\ndiff --git a/refs.h b/refs.h\nindex b94e1ac..796e396 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);\n /** Release any lock taken but not written. **/\n extern void unlock_ref(struct ref_lock *lock);\n \n-/** Writes sha1 into the ref specified by the lock. **/\n-extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);\n-\n /** Setup reflog before using. **/\n int log_ref_setup(const char *refname, char *logfile, int bufsize);\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241710","messageId":"1400174999-26786-30-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 29/44] refs.c: make lock_ref_sha1 static","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:44Z","receivedAt":"2014-05-15T17:29:44Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"No external callers reference lock_ref_sha1 any more so lets declare it static.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 2 +-\n refs.h | 3 ---\n 2 files changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex e5729ad..600f9b3 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2126,7 +2126,7 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \treturn NULL;\n }\n \n-struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)\n+static struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)\n {\n \tchar refpath[PATH_MAX];\n \tif (check_refname_format(refname, 0))\ndiff --git a/refs.h b/refs.h\nindex 796e396..221632c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -132,9 +132,6 @@ extern int ref_exists(const char *);\n */\n extern int peel_ref(const char *refname, unsigned char *sha1);\n \n-/** Locks a \"refs/\" ref returning the lock on success and NULL on failure. **/\n-extern struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1);\n-\n /** Locks any ref (for 'HEAD' type refs). */\n #define REF_NODEREF\t0x01\n extern struct ref_lock *lock_any_ref_for_update(const char *refname,\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241736","messageId":"1400174999-26786-31-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:45Z","receivedAt":"2014-05-15T17:29:45Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Track the status of a transaction in a new status field. Check the field for\nsanity, i.e. that status must be OPEN when _commit/_create/_delete or\n_update is called or else die(BUG:...)\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 31 ++++++++++++++++++++++++++++++-\n 1 file changed, 30 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 600f9b3..f11f832 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3308,6 +3308,12 @@ struct ref_update {\n \tconst char refname[FLEX_ARRAY];\n };\n \n+enum ref_transaction_status {\n+\tREF_TRANSACTION_OPEN = 0,\n+\tREF_TRANSACTION_CLOSED = 1,\n+\tREF_TRANSACTION_ERROR = 2,\n+};\n+\n /*\n * Data structure for holding a reference transaction, which can\n * consist of checks and updates to multiple references, carried out\n@@ -3317,6 +3323,7 @@ struct ref_transaction {\n \tstruct ref_update **updates;\n \tsize_t alloc;\n \tsize_t nr;\n+\tenum ref_transaction_status status;\n };\n \n struct ref_transaction *ref_transaction_begin(void)\n@@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \n void ref_transaction_rollback(struct ref_transaction *transaction)\n {\n+\tif (!transaction)\n+\t\treturn;\n+\n+\ttransaction->status = REF_TRANSACTION_ERROR;\n+\n \tref_transaction_free(transaction);\n }\n \n@@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tif (have_old && !old_sha1)\n \t\tdie(\"BUG: have_old is true but old_sha1 is NULL\");\n \n+\tif (transaction->status != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: update on transaction that is not open\");\n+\n \tupdate = add_update(transaction, refname);\n \thashcpy(update->new_sha1, new_sha1);\n \tupdate->flags = flags;\n@@ -3385,6 +3400,9 @@ int ref_transaction_create(struct ref_transaction *transaction,\n \tif (!new_sha1 || is_null_sha1(new_sha1))\n \t\tdie(\"BUG: create ref with null new_sha1\");\n \n+\tif (transaction->status != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: create on transaction that is not open\");\n+\n \tupdate = add_update(transaction, refname);\n \n \thashcpy(update->new_sha1, new_sha1);\n@@ -3404,6 +3422,9 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n \tif (have_old && !old_sha1)\n \t\tdie(\"BUG: have_old is true but old_sha1 is NULL\");\n \n+\tif (transaction->status != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: delete on transaction that is not open\");\n+\n \tupdate = add_update(transaction, refname);\n \tupdate->flags = flags;\n \tupdate->have_old = have_old;\n@@ -3474,8 +3495,13 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n \n-\tif (!n)\n+\tif (transaction->status != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: commit on transaction that is not open\");\n+\n+\tif (!n) {\n+\t\ttransaction->status = REF_TRANSACTION_CLOSED;\n \t\treturn 0;\n+\t}\n \n \t/* Allocate work space */\n \tdelnames = xmalloc(sizeof(*delnames) * n);\n@@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tclear_loose_ref_cache(&ref_cache);\n \n cleanup:\n+\ttransaction->status = ret ? REF_TRANSACTION_ERROR\n+\t : REF_TRANSACTION_CLOSED;\n+\n \tfor (i = 0; i < n; i++)\n \t\tif (updates[i]->lock)\n \t\t\tunlock_ref(updates[i]->lock);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241716","messageId":"1400174999-26786-32-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 31/44] refs.c: remove the update_ref_lock function","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:46Z","receivedAt":"2014-05-15T17:29:46Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Since we now only call update_ref_lock with onerr==QUIET_ON_ERR we no longer\nneed this function and can replace it with just calling lock_any_ref_for_update\ndirectly.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 30 ++++++------------------------\n 1 file changed, 6 insertions(+), 24 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f11f832..07377ad 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3255,24 +3255,6 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)\n \treturn retval;\n }\n \n-static struct ref_lock *update_ref_lock(const char *refname,\n-\t\t\t\t\tconst unsigned char *oldval,\n-\t\t\t\t\tint flags, int *type_p,\n-\t\t\t\t\tenum action_on_err onerr)\n-{\n-\tstruct ref_lock *lock;\n-\tlock = lock_any_ref_for_update(refname, oldval, flags, type_p);\n-\tif (!lock) {\n-\t\tconst char *str = \"Cannot lock the ref '%s'.\";\n-\t\tswitch (onerr) {\n-\t\tcase UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;\n-\t\tcase UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;\n-\t\tcase UPDATE_REFS_QUIET_ON_ERR: break;\n-\t\t}\n-\t}\n-\treturn lock;\n-}\n-\n static int update_ref_write(const char *action, const char *refname,\n \t\t\t const unsigned char *sha1, struct ref_lock *lock,\n \t\t\t struct strbuf *err, enum action_on_err onerr)\n@@ -3516,12 +3498,12 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n-\t\tupdate->lock = update_ref_lock(update->refname,\n-\t\t\t\t\t (update->have_old ?\n-\t\t\t\t\t\tupdate->old_sha1 : NULL),\n-\t\t\t\t\t update->flags,\n-\t\t\t\t\t &update->type,\n-\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR);\n+\t\tupdate->lock = lock_any_ref_for_update(update->refname,\n+\t\t\t\t\t\t (update->have_old ?\n+\t\t\t\t\t\t\tupdate->old_sha1 :\n+\t\t\t\t\t\t\tNULL),\n+\t\t\t\t\t\t update->flags,\n+\t\t\t\t\t\t &update->type);\n \t\tif (!update->lock) {\n \t\t\tif (err)\n \t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241733","messageId":"1400174999-26786-33-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 32/44] refs.c: remove the update_ref_write function","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:47Z","receivedAt":"2014-05-15T17:29:47Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Since we only call update_ref_write from a single place and we only call it\nwith onerr==QUIET_ON_ERR we can just as well get rid of it and just call\nwrite_ref_sha1 directly.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 35 +++++++++--------------------------\n 1 file changed, 9 insertions(+), 26 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 07377ad..7aa5512 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3255,25 +3255,6 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)\n \treturn retval;\n }\n \n-static int update_ref_write(const char *action, const char *refname,\n-\t\t\t const unsigned char *sha1, struct ref_lock *lock,\n-\t\t\t struct strbuf *err, enum action_on_err onerr)\n-{\n-\tif (write_ref_sha1(lock, sha1, action) < 0) {\n-\t\tconst char *str = \"Cannot update the ref '%s'.\";\n-\t\tif (err)\n-\t\t\tstrbuf_addf(err, str, refname);\n-\n-\t\tswitch (onerr) {\n-\t\tcase UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;\n-\t\tcase UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;\n-\t\tcase UPDATE_REFS_QUIET_ON_ERR: break;\n-\t\t}\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n /**\n * Information needed for a single ref update. Set new_sha1 to the\n * new value or to zero to delete the ref. To check the old value\n@@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \n \t\tif (!is_null_sha1(update->new_sha1)) {\n-\t\t\tret = update_ref_write(msg,\n-\t\t\t\t\t update->refname,\n-\t\t\t\t\t update->new_sha1,\n-\t\t\t\t\t update->lock, err,\n-\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR);\n-\t\t\tupdate->lock = NULL; /* freed by update_ref_write */\n-\t\t\tif (ret)\n+\t\t\tret = write_ref_sha1(update->lock, update->new_sha1,\n+\t\t\t\t\t msg);\n+\t\t\tupdate->lock = NULL; /* freed by write_ref_sha1 */\n+\t\t\tif (ret) {\n+\t\t\t\tconst char *str = \"Cannot update the ref '%s'.\";\n+\n+\t\t\t\tif (err)\n+\t\t\t\t\tstrbuf_addf(err, str, update->refname);\n \t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \t}\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241732","messageId":"1400174999-26786-34-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 33/44] refs.c: remove lock_ref_sha1","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:48Z","receivedAt":"2014-05-15T17:29:48Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"lock_ref_sha1 was only called from one place in refc.c and only provided\na check that the refname was sane before adding back the initial \"refs/\"\npart of the ref path name, the initial \"refs/\" that this caller had already\nstripped off before calling lock_ref_sha1.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 15 +++++----------\n 1 file changed, 5 insertions(+), 10 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 7aa5512..f1ef940 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2126,15 +2126,6 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \treturn NULL;\n }\n \n-static struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)\n-{\n-\tchar refpath[PATH_MAX];\n-\tif (check_refname_format(refname, 0))\n-\t\treturn NULL;\n-\tstrcpy(refpath, mkpath(\"refs/%s\", refname));\n-\treturn lock_ref_sha1_basic(refpath, old_sha1, 0, NULL);\n-}\n-\n struct ref_lock *lock_any_ref_for_update(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n \t\t\t\t\t int flags, int *type_p)\n@@ -2335,8 +2326,12 @@ static void try_remove_empty_parents(char *name)\n /* make sure nobody touched the ref, and unlink */\n static void prune_ref(struct ref_to_prune *r)\n {\n-\tstruct ref_lock *lock = lock_ref_sha1(r->name + 5, r->sha1);\n+\tstruct ref_lock *lock;\n+\n+\tif (check_refname_format(r->name + 5, 0))\n+\t\treturn;\n \n+\tlock = lock_ref_sha1_basic(r->name, r->sha1, 0, NULL);\n \tif (lock) {\n \t\tunlink_or_warn(git_path(\"%s\", r->name));\n \t\tunlock_ref(lock);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241722","messageId":"1400174999-26786-35-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:49Z","receivedAt":"2014-05-15T17:29:49Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change prune_ref to delete the ref using a ref transaction. To do this we also\nneed to add a new flag REF_ISPRUNING that will tell the transaction that we\ndo not want to delete this ref from the packed refs. This flag is private to\nrefs.c and not exposed to external callers.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f1ef940..88ab012 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -29,6 +29,11 @@ static inline int bad_ref_char(int ch)\n \treturn 0;\n }\n \n+/** Used as a flag to ref_transaction_delete when a loose ref is beeing\n+ * pruned.\n+ */\n+#define REF_ISPRUNING\t0x0100\n+\n /*\n * Try to read one refname component from the front of refname. Return\n * the length of the component found, or -1 if the component is not\n@@ -2326,17 +2331,24 @@ static void try_remove_empty_parents(char *name)\n /* make sure nobody touched the ref, and unlink */\n static void prune_ref(struct ref_to_prune *r)\n {\n-\tstruct ref_lock *lock;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n \n \tif (check_refname_format(r->name + 5, 0))\n \t\treturn;\n \n-\tlock = lock_ref_sha1_basic(r->name, r->sha1, 0, NULL);\n-\tif (lock) {\n-\t\tunlink_or_warn(git_path(\"%s\", r->name));\n-\t\tunlock_ref(lock);\n-\t\ttry_remove_empty_parents(r->name);\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_delete(transaction, r->name, r->sha1,\n+\t\t\t\t REF_ISPRUNING, 1) ||\n+\t ref_transaction_commit(transaction, NULL, &err)) {\n+\t\tref_transaction_rollback(transaction);\n+\t\twarning(\"prune_ref: %s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn;\n \t}\n+\tref_transaction_free(transaction);\n+\ttry_remove_empty_parents(r->name);\n }\n \n static void prune_refs(struct ref_to_prune *r)\n@@ -3512,9 +3524,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \n \t\tif (update->lock) {\n-\t\t\tdelnames[delnum++] = update->lock->ref_name;\n \t\t\tret |= delete_ref_loose(update->lock, update->type,\n \t\t\t\t\t\terr);\n+\t\t\tif (!(update->flags & REF_ISPRUNING))\n+\t\t\t\tdelnames[delnum++] = update->lock->ref_name;\n \t\t}\n \t}\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241720","messageId":"1400174999-26786-36-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 35/44] refs.c: make delete_ref use a transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:50Z","receivedAt":"2014-05-15T17:29:50Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change delete_ref to use a ref transaction for the deletion. At the same time\nsince we no longer have any callers of repack_without_ref we can now delete\nthis function.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 31 ++++++++++---------------------\n 1 file changed, 10 insertions(+), 21 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 88ab012..e7bd95d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2488,11 +2488,6 @@ static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n \treturn ret;\n }\n \n-static int repack_without_ref(const char *refname)\n-{\n-\treturn repack_without_refs(&refname, 1, NULL);\n-}\n-\n static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n {\n \tif (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {\n@@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n \n int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n {\n-\tstruct ref_lock *lock;\n-\tint ret = 0, flag = 0;\n+\tstruct ref_transaction *transaction;\n \n-\tlock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);\n-\tif (!lock)\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_delete(transaction, refname, sha1, delopt,\n+\t\t\t\t sha1 && !is_null_sha1(sha1)) ||\n+\t ref_transaction_commit(transaction, NULL, NULL)) {\n+\t\tref_transaction_rollback(transaction);\n \t\treturn 1;\n-\tret |= delete_ref_loose(lock, flag, NULL);\n-\n-\t/* removing the loose one could have resurrected an earlier\n-\t * packed one. Also, if it was not loose we need to repack\n-\t * without it.\n-\t */\n-\tret |= repack_without_ref(lock->ref_name);\n-\n-\tunlink_or_warn(git_path(\"logs/%s\", lock->ref_name));\n-\tclear_loose_ref_cache(&ref_cache);\n-\tunlock_ref(lock);\n-\treturn ret;\n+\t}\n+\tref_transaction_free(transaction);\n+\treturn 0;\n }\n \n /*\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241719","messageId":"1400174999-26786-37-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:51Z","receivedAt":"2014-05-15T17:29:51Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change the reference transactions so that we pass the reflog message\nthrough to the create/delete/update function instead of the commit message.\nThis allows for individual messages for each change in a multi ref\ntransaction.\n\nSigned-off-by: Ronnie Sahlberg \n---\n branch.c | 6 +++---\n builtin/commit.c | 4 ++--\n builtin/fetch.c | 10 ++++++++--\n builtin/receive-pack.c | 4 ++--\n builtin/replace.c | 4 ++--\n builtin/tag.c | 4 ++--\n builtin/update-ref.c | 13 +++++++------\n fast-import.c | 8 ++++----\n refs.c | 34 +++++++++++++++++++++-------------\n refs.h | 8 ++++----\n sequencer.c | 4 ++--\n walker.c | 6 +++---\n 12 files changed, 60 insertions(+), 45 deletions(-)\n\ndiff --git a/branch.c b/branch.c\nindex 8e58908..74d55e7 100644\n--- a/branch.c\n+++ b/branch.c\n@@ -300,9 +300,9 @@ void create_branch(const char *head,\n \n \t\ttransaction = ref_transaction_begin();\n \t\tif (!transaction ||\n-\t\t\tref_transaction_update(transaction, ref.buf, sha1,\n-\t\t\t\t\t null_sha1, 0, !forcing) ||\n-\t\t\tref_transaction_commit(transaction, msg, &err))\n+\t\t ref_transaction_update(transaction, ref.buf, sha1,\n+\t\t\t\t\t null_sha1, 0, !forcing, msg) ||\n+\t\t ref_transaction_commit(transaction, &err))\n \t\t\t\tdie_errno(_(\"%s: failed to write ref: %s\"),\n \t\t\t\t\t ref.buf, err.buf);\n \t\tref_transaction_free(transaction);\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 16fadbb..799a59f 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1721,8 +1721,8 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t ref_transaction_update(transaction, \"HEAD\", sha1,\n \t\t\t\t current_head ?\n \t\t\t\t current_head->object.sha1 : NULL,\n-\t\t\t\t 0, !!current_head) ||\n-\t ref_transaction_commit(transaction, sb.buf, &err)) {\n+\t\t\t\t 0, !!current_head, sb.buf) ||\n+\t ref_transaction_commit(transaction, &err)) {\n \t\trollback_index_files();\n \t\tdie(\"%s\", err.buf);\n \t}\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5dbd0f0..3a849b0 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -374,11 +374,17 @@ static int s_update_ref(const char *action,\n \t\t\tstruct ref *ref,\n \t\t\tint check_old)\n {\n+\tchar msg[1024];\n+\tchar *rla = getenv(\"GIT_REFLOG_ACTION\");\n+\n \tif (dry_run)\n \t\treturn 0;\n+\tif (!rla)\n+\t\trla = default_rla.buf;\n+\tsnprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n \n \tif (ref_transaction_update(transaction, ref->name, ref->new_sha1,\n-\t\t\t ref->old_sha1, 0, check_old))\n+\t\t\t\t ref->old_sha1, 0, check_old, msg))\n \t\treturn STORE_REF_ERROR_OTHER;\n \n \treturn 0;\n@@ -670,7 +676,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\t\t}\n \t\t}\n \t}\n-\tif (ref_transaction_commit(transaction, \"fetch_ref transaction\", NULL))\n+\tif (ref_transaction_commit(transaction, NULL))\n \t\trc |= errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n \t\t STORE_REF_ERROR_OTHER;\n \tref_transaction_free(transaction);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d580176..991c659 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -582,7 +582,7 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n \t\t\treturn \"shallow error\";\n \n \t\tif (ref_transaction_update(transaction, namespaced_name,\n-\t\t\t\t\t new_sha1, old_sha1, 0, 1))\n+\t\t\t\t\t new_sha1, old_sha1, 0, 1, \"push\"))\n \t\t\treturn \"failed to update\";\n \t\treturn NULL; /* good */\n \t}\n@@ -823,7 +823,7 @@ static void execute_commands(struct command *commands,\n \t\t\tchecked_connectivity = 0;\n \t\t}\n \t}\n-\tif (ref_transaction_commit(transaction, \"push\", &err))\n+\tif (ref_transaction_commit(transaction, &err))\n \t\terror(\"%s\", err.buf);\n \tref_transaction_free(transaction);\n \tstrbuf_release(&err);\ndiff --git a/builtin/replace.c b/builtin/replace.c\nindex 91354a7..0c46b3a 100644\n--- a/builtin/replace.c\n+++ b/builtin/replace.c\n@@ -160,8 +160,8 @@ static int replace_object_sha1(const char *object_ref,\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_update(transaction, ref, repl, prev,\n-\t\t\t\t 0, !is_null_sha1(prev)) ||\n-\t ref_transaction_commit(transaction, NULL, &err))\n+\t\t\t\t 0, !is_null_sha1(prev), NULL) ||\n+\t ref_transaction_commit(transaction, &err))\n \t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n \n \tref_transaction_free(transaction);\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex fbd2989..2cc260f 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -705,8 +705,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_update(transaction, ref.buf, object, prev,\n-\t\t\t\t 0, !is_null_sha1(prev)) ||\n-\t ref_transaction_commit(transaction, NULL, &err))\n+\t\t\t\t 0, !is_null_sha1(prev), NULL) ||\n+\t ref_transaction_commit(transaction, &err))\n \t\tdie(_(\"%s: cannot update the ref: %s\"), ref.buf, err.buf);\n \tref_transaction_free(transaction);\n \tif (force && !is_null_sha1(prev) && hashcmp(prev, object))\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex b5f4731..bd7e96f 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -16,6 +16,7 @@ static struct ref_transaction *transaction;\n \n static char line_termination = '\\n';\n static int update_flags;\n+static const char *msg;\n \n /*\n * Parse one whitespace- or NUL-terminated, possibly C-quoted argument\n@@ -198,7 +199,7 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)\n \t\tdie(\"update %s: extra input: %s\", refname, next);\n \n \tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n-\t\t\t\t update_flags, have_old))\n+\t\t\t\t update_flags, have_old, msg))\n \t\tdie(\"update %s: failed\", refname);\n \n \tupdate_flags = 0;\n@@ -226,7 +227,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)\n \t\tdie(\"create %s: extra input: %s\", refname, next);\n \n \tif (ref_transaction_create(transaction, refname, new_sha1,\n-\t\t\t\t update_flags))\n+\t\t\t\t update_flags, msg))\n \t\tdie(\"cannot create ref '%s'\", refname);\n \n \tupdate_flags = 0;\n@@ -258,7 +259,7 @@ static const char *parse_cmd_delete(struct strbuf *input, const char *next)\n \t\tdie(\"delete %s: extra input: %s\", refname, next);\n \n \tif (ref_transaction_delete(transaction, refname, old_sha1,\n-\t\t\t\t update_flags, have_old))\n+\t\t\t\t update_flags, have_old, msg))\n \t\tdie(\"failed transaction delete for %s\", refname);\n \n \tupdate_flags = 0;\n@@ -291,7 +292,7 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)\n \t\tdie(\"verify %s: extra input: %s\", refname, next);\n \n \tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n-\t\t\t\t update_flags, have_old))\n+\t\t\t\t update_flags, have_old, msg))\n \t\tdie(\"failed transaction update for %s\", refname);\n \n \tupdate_flags = 0;\n@@ -344,7 +345,7 @@ static void update_refs_stdin(void)\n \n int cmd_update_ref(int argc, const char **argv, const char *prefix)\n {\n-\tconst char *refname, *oldval, *msg = NULL;\n+\tconst char *refname, *oldval;\n \tunsigned char sha1[20], oldsha1[20];\n \tint delete = 0, no_deref = 0, read_stdin = 0, end_null = 0, flags = 0;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -371,7 +372,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)\n \t\tif (end_null)\n \t\t\tline_termination = '\\0';\n \t\tupdate_refs_stdin();\n-\t\tif (ref_transaction_commit(transaction, msg, &err))\n+\t\tif (ref_transaction_commit(transaction, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\tref_transaction_free(transaction);\n \t\treturn 0;\ndiff --git a/fast-import.c b/fast-import.c\nindex 5587cf6..099e71b 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1710,8 +1710,8 @@ static int update_branch(struct branch *b)\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_update(transaction, b->name, b->sha1, old_sha1,\n-\t\t\t\t 0, 1) ||\n-\t ref_transaction_commit(transaction, msg, &err)) {\n+\t\t\t\t 0, 1, msg) ||\n+\t ref_transaction_commit(transaction, &err)) {\n \t\tref_transaction_rollback(transaction);\n \t\terror(\"Unable to update branch %s: %s\", b->name, err.buf);\n \t\tstrbuf_release(&err);\n@@ -1745,10 +1745,10 @@ static void dump_tags(void)\n \t\tsprintf(ref_name, \"refs/tags/%s\", t->name);\n \n \t\tif (ref_transaction_update(transaction, ref_name, t->sha1,\n-\t\t\t\t\t NULL, 0, 0))\n+\t\t\t\t\t NULL, 0, 0, msg))\n \t\t\tfailure |= error(\"Unable to update %s\", err.buf);\n \t}\n-\tif (failure || ref_transaction_commit(transaction, msg, &err))\n+\tif (failure || ref_transaction_commit(transaction, &err))\n \t\tfailure |= error(\"Unable to update %s\", err.buf);\n \tref_transaction_free(transaction);\n \tstrbuf_release(&err);\ndiff --git a/refs.c b/refs.c\nindex e7bd95d..1977640 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2340,8 +2340,8 @@ static void prune_ref(struct ref_to_prune *r)\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_delete(transaction, r->name, r->sha1,\n-\t\t\t\t REF_ISPRUNING, 1) ||\n-\t ref_transaction_commit(transaction, NULL, &err)) {\n+\t\t\t\t REF_ISPRUNING, 1, NULL) ||\n+\t ref_transaction_commit(transaction, &err)) {\n \t\tref_transaction_rollback(transaction);\n \t\twarning(\"prune_ref: %s\", err.buf);\n \t\tstrbuf_release(&err);\n@@ -2515,8 +2515,8 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_delete(transaction, refname, sha1, delopt,\n-\t\t\t\t sha1 && !is_null_sha1(sha1)) ||\n-\t ref_transaction_commit(transaction, NULL, NULL)) {\n+\t\t\t\t sha1 && !is_null_sha1(sha1), NULL) ||\n+\t ref_transaction_commit(transaction, NULL)) {\n \t\tref_transaction_rollback(transaction);\n \t\treturn 1;\n \t}\n@@ -3264,6 +3264,7 @@ struct ref_update {\n \tint have_old; /* 1 if old_sha1 is valid, 0 otherwise */\n \tstruct ref_lock *lock;\n \tint type;\n+\tconst char *msg;\n \tconst char refname[FLEX_ARRAY];\n };\n \n@@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \tif (!transaction)\n \t\treturn;\n \n-\tfor (i = 0; i < transaction->nr; i++)\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t free((char *)transaction->updates[i]->msg);\n \t\tfree(transaction->updates[i]);\n-\n+\t}\n \tfree(transaction->updates);\n \tfree(transaction);\n }\n@@ -3330,7 +3332,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *new_sha1,\n \t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old)\n+\t\t\t int flags, int have_old, const char *msg)\n {\n \tstruct ref_update *update;\n \n@@ -3346,13 +3348,15 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tupdate->have_old = have_old;\n \tif (have_old)\n \t\thashcpy(update->old_sha1, old_sha1);\n+\tif (msg)\n+\t\tupdate->msg = xstrdup(msg);\n \treturn 0;\n }\n \n int ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *new_sha1,\n-\t\t\t int flags)\n+\t\t\t int flags, const char *msg)\n {\n \tstruct ref_update *update;\n \n@@ -3368,13 +3372,15 @@ int ref_transaction_create(struct ref_transaction *transaction,\n \thashclr(update->old_sha1);\n \tupdate->flags = flags;\n \tupdate->have_old = 1;\n+\tif (msg)\n+\t\tupdate->msg = xstrdup(msg);\n \treturn 0;\n }\n \n int ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old)\n+\t\t\t int flags, int have_old, const char *msg)\n {\n \tstruct ref_update *update;\n \n@@ -3391,6 +3397,8 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n \t\tassert(!is_null_sha1(old_sha1));\n \t\thashcpy(update->old_sha1, old_sha1);\n \t}\n+\tif (msg)\n+\t\tupdate->msg = xstrdup(msg);\n \treturn 0;\n }\n \n@@ -3404,8 +3412,8 @@ int update_ref(const char *action, const char *refname,\n \tt = ref_transaction_begin();\n \tif (!t ||\n \t ref_transaction_update(t, refname, sha1, oldval, flags,\n-\t\t\t\t !!oldval) ||\n-\t ref_transaction_commit(t, action, &err)) {\n+\t\t\t\t !!oldval, action) ||\n+\t ref_transaction_commit(t, &err)) {\n \t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n \n \t\tref_transaction_rollback(t);\n@@ -3447,7 +3455,7 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,\n }\n \n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, struct strbuf *err)\n+\t\t\t struct strbuf *err)\n {\n \tint ret = 0, delnum = 0, i;\n \tconst char **delnames;\n@@ -3496,7 +3504,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \t\tif (!is_null_sha1(update->new_sha1)) {\n \t\t\tret = write_ref_sha1(update->lock, update->new_sha1,\n-\t\t\t\t\t msg);\n+\t\t\t\t\t update->msg);\n \t\t\tupdate->lock = NULL; /* freed by write_ref_sha1 */\n \t\t\tif (ret) {\n \t\t\t\tconst char *str = \"Cannot update the ref '%s'.\";\ndiff --git a/refs.h b/refs.h\nindex 221632c..0e6d416 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -244,7 +244,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *new_sha1,\n \t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old);\n+\t\t\t int flags, int have_old, const char *msg);\n \n /*\n * Add a reference creation to transaction. new_sha1 is the value\n@@ -258,7 +258,7 @@ int ref_transaction_update(struct ref_transaction *transaction,\n int ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *new_sha1,\n-\t\t\t int flags);\n+\t\t\t int flags, const char *msg);\n \n /*\n * Add a reference deletion to transaction. If have_old is true, then\n@@ -271,7 +271,7 @@ int ref_transaction_create(struct ref_transaction *transaction,\n int ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n \t\t\t const unsigned char *old_sha1,\n-\t\t\t int flags, int have_old);\n+\t\t\t int flags, int have_old, const char *msg);\n \n /*\n * Commit all of the changes that have been queued in transaction, as\n@@ -280,7 +280,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n * why the transaction failed. The string does not end in newline.\n */\n int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t const char *msg, struct strbuf *err);\n+\t\t\t struct strbuf *err);\n \n /*\n * Free an existing transaction.\ndiff --git a/sequencer.c b/sequencer.c\nindex 0cfdaf0..3a0ee09 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -285,8 +285,8 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,\n \ttransaction = ref_transaction_begin();\n \tif (!transaction ||\n \t ref_transaction_update(transaction, \"HEAD\", to, from,\n-\t\t\t\t 0, !unborn) ||\n-\t ref_transaction_commit(transaction, sb.buf, &err)) {\n+\t\t\t\t 0, !unborn, sb.buf) ||\n+\t ref_transaction_commit(transaction, &err)) {\n \t\tref_transaction_rollback(transaction);\n \t\terror(_(\"HEAD: Could not fast-forward: %s\\n\"), err.buf);\n \t\tstrbuf_release(&sb);\ndiff --git a/walker.c b/walker.c\nindex 6044ccf..c2a1266 100644\n--- a/walker.c\n+++ b/walker.c\n@@ -291,12 +291,12 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n \t\tsprintf(ref_name, \"refs/%s\", write_ref[i]);\n \t\tif (ref_transaction_update(transaction, ref_name,\n \t\t\t\t\t &sha1[20 * i], NULL,\n-\t\t\t\t\t 0, 0))\n+\t\t\t\t\t 0, 0,\n+\t\t\t\t\t msg ? msg : \"fetch (unknown)\"))\n \t\t\tgoto rollback_and_fail;\n \t}\n \n-\tif (ref_transaction_commit(transaction, msg ? msg : \"fetch (unknown)\",\n-\t\t\t\t &err)) {\n+\tif (ref_transaction_commit(transaction, &err)) {\n \t\terror(\"%s\", err.buf);\n \t\tgoto rollback_and_fail;\n \t}\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241728","messageId":"1400174999-26786-38-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 37/44] refs.c: pass NULL as *flags to read_ref_full","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:52Z","receivedAt":"2014-05-15T17:29:52Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"We call read_ref_full with a pointer to flags from rename_ref but since\nwe never actually use the returned flags we can just pass NULL here instead.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1977640..8105ee6 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2612,7 +2612,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \t\tgoto rollback;\n \t}\n \n-\tif (!read_ref_full(newrefname, sha1, 1, &flag) &&\n+\tif (!read_ref_full(newrefname, sha1, 1, NULL) &&\n \t delete_ref(newrefname, sha1, REF_NODEREF)) {\n \t\tif (errno==EISDIR) {\n \t\t\tif (remove_empty_directories(git_path(\"%s\", newrefname))) {\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241725","messageId":"1400174999-26786-39-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:53Z","receivedAt":"2014-05-15T17:29:53Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"This means that most loose refs will no longer be present after the rename\nwhich triggered a test failure since it assumes the file for an unrelated\nref would still be present after the rename.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 3 +++\n t/t3200-branch.sh | 2 +-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8105ee6..79d95db 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2607,6 +2607,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \t\treturn error(\"unable to move logfile logs/%s to \"TMP_RENAMED_LOG\": %s\",\n \t\t\toldrefname, strerror(errno));\n \n+\tif (pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE))\n+\t\treturn error(\"unable to pack refs\");\n+\n \tif (delete_ref(oldrefname, orig_sha1, REF_NODEREF)) {\n \t\terror(\"unable to delete old %s\", oldrefname);\n \t\tgoto rollback;\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex ac31b71..fafd38a 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '\n \tgit symbolic-ref refs/heads/master2 refs/heads/master &&\n \ttest_must_fail git branch -m master2 master3 &&\n \tgit symbolic-ref refs/heads/master2 &&\n-\ttest_path_is_file .git/refs/heads/master &&\n+\ttest_path_is_missing .git/refs/heads/master &&\n \ttest_path_is_missing .git/refs/heads/master3\n '\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241729","messageId":"1400174999-26786-40-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:54Z","receivedAt":"2014-05-15T17:29:54Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Move the check for check_refname_format from lock_any_ref_for_update\nto lock_ref_sha1_basic. At some later stage we will get rid of\nlock_any_ref_for_update completely.\n\nThis leaves lock_any_ref_for_updates as a no-op wrapper which could be removed.\nBut this wrapper is also called from an external caller and we will soon\nmake changes to the signature to lock_ref_sha1_basic that we do not want to\nexpose to that caller.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 79d95db..53fdbd1 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \tint missing = 0;\n \tint attempts_remaining = 3;\n \n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n+\t\treturn NULL;\n+\n \tlock = xcalloc(1, sizeof(struct ref_lock));\n \tlock->lock_fd = -1;\n \n@@ -2135,8 +2138,6 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n \t\t\t\t\t int flags, int *type_p)\n {\n-\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n-\t\treturn NULL;\n \treturn lock_ref_sha1_basic(refname, old_sha1, flags, type_p);\n }\n \n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241723","messageId":"1400174999-26786-41-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 40/44] refs.c: call lock_ref_sha1_basic directly from commit","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:55Z","receivedAt":"2014-05-15T17:29:55Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Skip using the lock_any_ref_for_update wrapper and call lock_ref_sha1_basic\ndirectly from the commit function.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 53fdbd1..69ad51d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3487,12 +3487,12 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n-\t\tupdate->lock = lock_any_ref_for_update(update->refname,\n-\t\t\t\t\t\t (update->have_old ?\n-\t\t\t\t\t\t\tupdate->old_sha1 :\n-\t\t\t\t\t\t\tNULL),\n-\t\t\t\t\t\t update->flags,\n-\t\t\t\t\t\t &update->type);\n+\t\tupdate->lock = lock_ref_sha1_basic(update->refname,\n+\t\t\t\t\t\t (update->have_old ?\n+\t\t\t\t\t\t update->old_sha1 :\n+\t\t\t\t\t\t NULL),\n+\t\t\t\t\t\t update->flags,\n+\t\t\t\t\t\t &update->type);\n \t\tif (!update->lock) {\n \t\t\tif (err)\n \t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241727","messageId":"1400174999-26786-42-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:56Z","receivedAt":"2014-05-15T17:29:56Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.\nThis flag indicates that the ref does not exist as a loose ref andf only as\na packed ref. If this is the case we then change the commit code so that\nwe skip taking out a lock file and we skip calling delete_ref_loose.\nCheck for this flag and die(BUG:...) if used with _update or _create.\n\nAt the start of the transaction, before we even start locking any refs,\nwe add all such REF_ISPACKONLY refs to delnames so that we have a list of\nall pack only refs that we will be deleting during this transaction.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 69ad51d..f458ff9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -33,6 +33,10 @@ static inline int bad_ref_char(int ch)\n * pruned.\n */\n #define REF_ISPRUNING\t0x0100\n+/** Deletion of a ref that only exists as a packed ref in which case we do not\n+ * need to lock the loose ref during the transaction.\n+ */\n+#define REF_ISPACKONLY\t0x0200\n \n /*\n * Try to read one refname component from the front of refname. Return\n@@ -3346,6 +3350,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \tif (transaction->status != REF_TRANSACTION_OPEN)\n \t\tdie(\"BUG: update on transaction that is not open\");\n \n+\tif (flags & REF_ISPACKONLY)\n+\t\tdie(\"BUG: REF_ISPACKONLY can not be used with updates\");\n+\n \tupdate = add_update(transaction, refname);\n \thashcpy(update->new_sha1, new_sha1);\n \tupdate->flags = flags;\n@@ -3370,6 +3377,9 @@ int ref_transaction_create(struct ref_transaction *transaction,\n \tif (transaction->status != REF_TRANSACTION_OPEN)\n \t\tdie(\"BUG: create on transaction that is not open\");\n \n+\tif (flags & REF_ISPACKONLY)\n+\t\tdie(\"BUG: REF_ISPACKONLY can not be used with creates\");\n+\n \tupdate = add_update(transaction, refname);\n \n \thashcpy(update->new_sha1, new_sha1);\n@@ -3483,10 +3493,20 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tif (ret)\n \t\tgoto cleanup;\n \n+\tfor (i = 0; i < n; i++) {\n+\t\tstruct ref_update *update = updates[i];\n+\n+\t\tif (update->flags & REF_ISPACKONLY)\n+\t\t\tdelnames[delnum++] = update->refname;\n+\t}\n+\n \t/* Acquire all locks while verifying old values */\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n+\t\tif (update->flags & REF_ISPACKONLY)\n+\t\t\tcontinue;\n+\n \t\tupdate->lock = lock_ref_sha1_basic(update->refname,\n \t\t\t\t\t\t (update->have_old ?\n \t\t\t\t\t\t update->old_sha1 :\n@@ -3524,6 +3544,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n+\t\tif (update->flags & REF_ISPACKONLY)\n+\t\t\tcontinue;\n+\n \t\tif (update->lock) {\n \t\t\tret |= delete_ref_loose(update->lock, update->type,\n \t\t\t\t\t\terr);\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241718","messageId":"1400174999-26786-43-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:57Z","receivedAt":"2014-05-15T17:29:57Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Allow passing a list of refs to skip checking to name_conflict_fn.\nThere are some conditions where we want to allow a temporary conflict and skip\nchecking those refs. For example if we have a transaction that\n1, guarantees that m is a packed refs and there is no loose ref for m\n2, the transaction will delete m from the packed ref\n3, the transaction will create conflicting m/m\n\nFor this case we want to be able to lock and create m/m since we know that the\nconflict is only transient. I.e. the conflict will be automatically resolved\nby the transaction when it deletes m.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 43 +++++++++++++++++++++++++++++++++----------\n 1 file changed, 33 insertions(+), 10 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f458ff9..c940509 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -798,11 +798,19 @@ struct name_conflict_cb {\n \tconst char *refname;\n \tconst char *oldrefname;\n \tconst char *conflicting_refname;\n+\tconst char **skip;\n+\tint skipnum;\n };\n \n static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n {\n \tstruct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;\n+\tint i;\n+\tfor(i = 0; i < data->skipnum; i++) {\n+\t\tif (!strcmp(entry->name, data->skip[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \tif (data->oldrefname && !strcmp(data->oldrefname, entry->name))\n \t\treturn 0;\n \tif (names_conflict(data->refname, entry->name)) {\n@@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n * conflicting with the name of an existing reference in dir. If\n * oldrefname is non-NULL, ignore potential conflicts with oldrefname\n * (e.g., because oldrefname is scheduled for deletion in the same\n- * operation).\n+ * operation). skip contains a list of refs we want to skip checking for\n+ * conflicts with. Refs may be skipped due to us knowing that it will\n+ * be deleted later during a transaction that deletes one reference and then\n+ * creates a new conflicting reference. For example a rename from m to m/m.\n */\n static int is_refname_available(const char *refname, const char *oldrefname,\n-\t\t\t\tstruct ref_dir *dir)\n+\t\t\t\tstruct ref_dir *dir,\n+\t\t\t\tconst char **skip, int skipnum)\n {\n \tstruct name_conflict_cb data;\n \tdata.refname = refname;\n \tdata.oldrefname = oldrefname;\n \tdata.conflicting_refname = NULL;\n+\tdata.skip = skip;\n+\tdata.skipnum = skipnum;\n \n \tsort_ref_dir(dir);\n \tif (do_for_each_entry_in_dir(dir, 0, name_conflict_fn, &data)) {\n@@ -2037,7 +2051,8 @@ int dwim_log(const char *str, int len, unsigned char *sha1, char **log)\n \n static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n-\t\t\t\t\t int flags, int *type_p)\n+\t\t\t\t\t int flags, int *type_p,\n+\t\t\t\t\t const char **skip, int skipnum)\n {\n \tchar *ref_file;\n \tconst char *orig_refname = refname;\n@@ -2084,7 +2099,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t * name is a proper prefix of our refname.\n \t */\n \tif (missing &&\n-\t !is_refname_available(refname, NULL, get_packed_refs(&ref_cache))) {\n+\t !is_refname_available(refname, NULL,\n+\t\t\t\t get_packed_refs(&ref_cache),\n+\t\t\t\t skip, skipnum)) {\n \t\tlast_errno = ENOTDIR;\n \t\tgoto error_return;\n \t}\n@@ -2142,7 +2159,7 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n \t\t\t\t\t int flags, int *type_p)\n {\n-\treturn lock_ref_sha1_basic(refname, old_sha1, flags, type_p);\n+\treturn lock_ref_sha1_basic(refname, old_sha1, flags, type_p, NULL, 0);\n }\n \n /*\n@@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \tint log = !lstat(git_path(\"logs/%s\", oldrefname), &loginfo);\n \tconst char *symref = NULL;\n \n+\tif (!strcmp(oldrefname, newrefname))\n+\t\treturn 0;\n+\n \tif (log && S_ISLNK(loginfo.st_mode))\n \t\treturn error(\"reflog for %s is a symlink\", oldrefname);\n \n@@ -2602,10 +2622,12 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \tif (!symref)\n \t\treturn error(\"refname %s not found\", oldrefname);\n \n-\tif (!is_refname_available(newrefname, oldrefname, get_packed_refs(&ref_cache)))\n+\tif (!is_refname_available(newrefname, oldrefname,\n+\t\t\t\t get_packed_refs(&ref_cache), NULL, 0))\n \t\treturn 1;\n \n-\tif (!is_refname_available(newrefname, oldrefname, get_loose_refs(&ref_cache)))\n+\tif (!is_refname_available(newrefname, oldrefname,\n+\t\t\t\t get_loose_refs(&ref_cache), NULL, 0))\n \t\treturn 1;\n \n \tif (log && rename(git_path(\"logs/%s\", oldrefname), git_path(TMP_RENAMED_LOG)))\n@@ -2638,7 +2660,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \n \tlogmoved = log;\n \n-\tlock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL);\n+\tlock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL, NULL, 0);\n \tif (!lock) {\n \t\terror(\"unable to lock %s for update\", newrefname);\n \t\tgoto rollback;\n@@ -2653,7 +2675,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \treturn 0;\n \n rollback:\n-\tlock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL);\n+\tlock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL, NULL, 0);\n \tif (!lock) {\n \t\terror(\"unable to lock %s for rollback\", oldrefname);\n \t\tgoto rollbacklog;\n@@ -3512,7 +3534,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t\t update->old_sha1 :\n \t\t\t\t\t\t NULL),\n \t\t\t\t\t\t update->flags,\n-\t\t\t\t\t\t &update->type);\n+\t\t\t\t\t\t &update->type,\n+\t\t\t\t\t\t delnames, delnum);\n \t\tif (!update->lock) {\n \t\t\tif (err)\n \t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241726","messageId":"1400174999-26786-44-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 43/44] refs.c: make rename_ref use a transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:58Z","receivedAt":"2014-05-15T17:29:58Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Change rename_ref to use a single transaction to perform the ref rename.\n\nSigned-off-by: Ronnie Sahlberg \n---\n refs.c | 73 ++++++++++++++++++------------------------------------------------\n 1 file changed, 20 insertions(+), 53 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex c940509..336df40 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2602,9 +2602,10 @@ static int rename_tmp_log(const char *newrefname)\n \n int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)\n {\n-\tunsigned char sha1[20], orig_sha1[20];\n-\tint flag = 0, logmoved = 0;\n-\tstruct ref_lock *lock;\n+\tunsigned char sha1[20];\n+\tint flag = 0;\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n \tstruct stat loginfo;\n \tint log = !lstat(git_path(\"logs/%s\", oldrefname), &loginfo);\n \tconst char *symref = NULL;\n@@ -2615,7 +2616,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \tif (log && S_ISLNK(loginfo.st_mode))\n \t\treturn error(\"reflog for %s is a symlink\", oldrefname);\n \n-\tsymref = resolve_ref_unsafe(oldrefname, orig_sha1, 1, &flag);\n+\tsymref = resolve_ref_unsafe(oldrefname, sha1, 1, &flag);\n \tif (flag & REF_ISSYMREF)\n \t\treturn error(\"refname %s is a symbolic ref, renaming it is not supported\",\n \t\t\toldrefname);\n@@ -2637,62 +2638,28 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n \tif (pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE))\n \t\treturn error(\"unable to pack refs\");\n \n-\tif (delete_ref(oldrefname, orig_sha1, REF_NODEREF)) {\n-\t\terror(\"unable to delete old %s\", oldrefname);\n-\t\tgoto rollback;\n-\t}\n-\n-\tif (!read_ref_full(newrefname, sha1, 1, NULL) &&\n-\t delete_ref(newrefname, sha1, REF_NODEREF)) {\n-\t\tif (errno==EISDIR) {\n-\t\t\tif (remove_empty_directories(git_path(\"%s\", newrefname))) {\n-\t\t\t\terror(\"Directory not empty: %s\", newrefname);\n-\t\t\t\tgoto rollback;\n-\t\t\t}\n-\t\t} else {\n-\t\t\terror(\"unable to delete existing %s\", newrefname);\n-\t\t\tgoto rollback;\n-\t\t}\n+\ttransaction = ref_transaction_begin();\n+\tif (!transaction ||\n+\t ref_transaction_delete(transaction, oldrefname, sha1,\n+\t\t\t\t REF_NODEREF | REF_ISPACKONLY,\n+\t\t\t\t 1, NULL) ||\n+\t ref_transaction_update(transaction, newrefname, sha1,\n+\t\t\t\t NULL, 0, 0, logmsg) ||\n+\t ref_transaction_commit(transaction, &err)) {\n+\t\tref_transaction_rollback(transaction);\n+\t\terror(\"rename_ref failed: %s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\tgoto rollbacklog;\n \t}\n+\tref_transaction_free(transaction);\n \n \tif (log && rename_tmp_log(newrefname))\n-\t\tgoto rollback;\n-\n-\tlogmoved = log;\n-\n-\tlock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL, NULL, 0);\n-\tif (!lock) {\n-\t\terror(\"unable to lock %s for update\", newrefname);\n-\t\tgoto rollback;\n-\t}\n-\tlock->force_write = 1;\n-\thashcpy(lock->old_sha1, orig_sha1);\n-\tif (write_ref_sha1(lock, orig_sha1, logmsg)) {\n-\t\terror(\"unable to write current sha1 into %s\", newrefname);\n-\t\tgoto rollback;\n-\t}\n-\n-\treturn 0;\n-\n- rollback:\n-\tlock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL, NULL, 0);\n-\tif (!lock) {\n-\t\terror(\"unable to lock %s for rollback\", oldrefname);\n \t\tgoto rollbacklog;\n-\t}\n \n-\tlock->force_write = 1;\n-\tflag = log_all_ref_updates;\n-\tlog_all_ref_updates = 0;\n-\tif (write_ref_sha1(lock, orig_sha1, NULL))\n-\t\terror(\"unable to write current sha1 into %s\", oldrefname);\n-\tlog_all_ref_updates = flag;\n+\treturn 0;\n \n rollbacklog:\n-\tif (logmoved && rename(git_path(\"logs/%s\", newrefname), git_path(\"logs/%s\", oldrefname)))\n-\t\terror(\"unable to restore logfile %s from %s: %s\",\n-\t\t\toldrefname, newrefname, strerror(errno));\n-\tif (!logmoved && log &&\n+\tif (log &&\n \t rename(git_path(TMP_RENAMED_LOG), git_path(\"logs/%s\", oldrefname)))\n \t\terror(\"unable to restore logfile %s from \"TMP_RENAMED_LOG\": %s\",\n \t\t\toldrefname, strerror(errno));\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241724","messageId":"1400174999-26786-45-git-send-email-sahlberg@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"[PATCH v8 44/44] refs.c: remove forward declaration of write_ref_sha1","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T17:29:59Z","receivedAt":"2014-05-15T17:29:59Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \n---\n refs.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 336df40..ed93b75 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -260,8 +260,6 @@ struct ref_entry {\n };\n \n static void read_loose_refs(const char *dirname, struct ref_dir *dir);\n-static int write_ref_sha1(struct ref_lock *lock,\n-\t\t\t const unsigned char *sha1, const char *logmsg);\n \n static struct ref_dir *get_ref_dir(struct ref_entry *entry)\n {\n-- \n2.0.0.rc3.477.g0f8edf7\n"},{"id":"241755","messageId":"20140515180618.GA26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T18:06:18Z","receivedAt":"2014-05-15T18:06:18Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This patch series is based on next and expands on the transaction API.\n\nThanks. Will pick up in v8 where I left off with v6.\n\nApplies with just one minor conflict on top of a merge of\nmh/ref-transaction, rs/ref-update-check-errors-early, and\nrs/reflog-exists. Here's an interdiff against version 6 for those\nfollowing along.\n\n> Version 8:\n> - Updates after review by JN\n> - Improve commit messages\n> - Add a patch that adds an err argument to repack_without_refs\n> - Add a patch that adds an err argument to delete_loose_ref\n> - Better document that a _update/_delete/_create failure means the whole\n> transaction has failed and needs rollback.\n>\n> Version 7:\n> - Updated commit messages per JNs review comments.\n> - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not\n> exposed through refs.h\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 0f4e1fc..07ccc97 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1684,7 +1684,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t\t\t 0, !!current_head, sb.buf) ||\n \t ref_transaction_commit(transaction, &err)) {\n \t\trollback_index_files();\n-\t\tdie(_(\"HEAD: cannot update ref: %s\"), err.buf);\n+\t\tdie(\"%s\", err.buf);\n \t}\n \tref_transaction_free(transaction);\n \ndiff --git a/builtin/replace.c b/builtin/replace.c\nindex 47c360c..952b589 100644\n--- a/builtin/replace.c\n+++ b/builtin/replace.c\n@@ -163,7 +163,7 @@ static int replace_object(const char *object_ref, const char *replace_ref,\n \t ref_transaction_update(transaction, ref, repl, prev,\n \t\t\t\t 0, !is_null_sha1(prev), NULL) ||\n \t ref_transaction_commit(transaction, &err))\n-\t\tdie(_(\"%s: failed to replace ref: %s\"), ref, err.buf);\n+\t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n \n \tref_transaction_free(transaction);\n \treturn 0;\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex c5aff92..bd7e96f 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -228,7 +228,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)\n \n \tif (ref_transaction_create(transaction, refname, new_sha1,\n \t\t\t\t update_flags, msg))\n-\t\tdie(\"failed transaction create for %s\", refname);\n+\t\tdie(\"cannot create ref '%s'\", refname);\n \n \tupdate_flags = 0;\n \tfree(refname);\ndiff --git a/refs.c b/refs.c\nindex 302a2b3..ed93b75 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -29,6 +29,15 @@ static inline int bad_ref_char(int ch)\n \treturn 0;\n }\n \n+/** Used as a flag to ref_transaction_delete when a loose ref is beeing\n+ * pruned.\n+ */\n+#define REF_ISPRUNING\t0x0100\n+/** Deletion of a ref that only exists as a packed ref in which case we do not\n+ * need to lock the loose ref during the transaction.\n+ */\n+#define REF_ISPACKONLY\t0x0200\n+\n /*\n * Try to read one refname component from the front of refname. Return\n * the length of the component found, or -1 if the component is not\n@@ -2447,12 +2456,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n \treturn 0;\n }\n \n-static int repack_without_refs(const char **refnames, int n)\n+static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n {\n \tstruct ref_dir *packed;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *ref_to_delete;\n-\tint i, removed = 0;\n+\tint i, ret, removed = 0;\n \n \t/* Look for a packed ref */\n \tfor (i = 0; i < n; i++)\n@@ -2465,6 +2474,9 @@ static int repack_without_refs(const char **refnames, int n)\n \n \tif (lock_packed_refs(0)) {\n \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n+\t\tif (err)\n+\t\t\tstrbuf_addf(err, \"cannot delete '%s' from packed refs\",\n+\t\t\t\t refnames[i]);\n \t\treturn error(\"cannot delete '%s' from packed refs\", refnames[i]);\n \t}\n \tpacked = get_packed_refs(&ref_cache);\n@@ -2490,20 +2502,28 @@ static int repack_without_refs(const char **refnames, int n)\n \t}\n \n \t/* Write what remains */\n-\treturn commit_packed_refs();\n+\tret = commit_packed_refs();\n+\tif (ret && err)\n+\t\tstrbuf_addf(err, \"unable to overwrite old ref-pack file\");\n+\treturn ret;\n }\n \n-static int delete_ref_loose(struct ref_lock *lock, int flag)\n+static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n {\n \tif (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {\n \t\t/* loose */\n-\t\tint err, i = strlen(lock->lk->filename) - 5; /* .lock */\n+\t\tint res, i = strlen(lock->lk->filename) - 5; /* .lock */\n \n \t\tlock->lk->filename[i] = 0;\n-\t\terr = unlink_or_warn(lock->lk->filename);\n+\t\tres = unlink_or_warn(lock->lk->filename);\n \t\tlock->lk->filename[i] = '.';\n-\t\tif (err && errno != ENOENT)\n+\t\tif (res && errno != ENOENT) {\n+\t\t\tif (err)\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t \"failed to delete loose ref '%s'\",\n+\t\t\t\t\t lock->lk->filename);\n \t\t\treturn 1;\n+\t\t}\n \t}\n \treturn 0;\n }\n@@ -3483,7 +3503,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t\t delnames, delnum);\n \t\tif (!update->lock) {\n \t\t\tif (err)\n-\t\t\t\tstrbuf_addf(err ,\"Cannot lock the ref '%s'.\",\n+\t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n \t\t\t\t\t update->refname);\n \t\t\tret = 1;\n \t\t\tgoto cleanup;\n@@ -3516,13 +3536,14 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\tcontinue;\n \n \t\tif (update->lock) {\n-\t\t\tret |= delete_ref_loose(update->lock, update->type);\n+\t\t\tret |= delete_ref_loose(update->lock, update->type,\n+\t\t\t\t\t\terr);\n \t\t\tif (!(update->flags & REF_ISPRUNING))\n \t\t\t\tdelnames[delnum++] = update->lock->ref_name;\n \t\t}\n \t}\n \n-\tret |= repack_without_refs(delnames, delnum);\n+\tret |= repack_without_refs(delnames, delnum, err);\n \tfor (i = 0; i < delnum; i++)\n \t\tunlink_or_warn(git_path(\"logs/%s\", delnames[i]));\n \tclear_loose_ref_cache(&ref_cache);\ndiff --git a/refs.h b/refs.h\nindex 03b4a7e..0e6d416 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -134,10 +134,6 @@ extern int peel_ref(const char *refname, unsigned char *sha1);\n \n /** Locks any ref (for 'HEAD' type refs). */\n #define REF_NODEREF\t0x01\n-/** Deleting a loose ref during prune */\n-#define REF_ISPRUNING\t0x02\n-/** Deletion of a ref that only exists as a packed ref */\n-#define REF_ISPACKONLY\t0x04\n extern struct ref_lock *lock_any_ref_for_update(const char *refname,\n \t\t\t\t\t\tconst unsigned char *old_sha1,\n \t\t\t\t\t\tint flags, int *type_p);\n@@ -240,6 +236,9 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n * be deleted. If have_old is true, then old_sha1 holds the value\n * that the reference should have had before the update, or zeros if\n * it must not have existed beforehand.\n+ * Function returns 0 on success and non-zero on failure. A failure to update\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n int ref_transaction_update(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n@@ -252,6 +251,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n * that the reference should have after the update; it must not be the\n * null SHA-1. It is verified that the reference does not exist\n * already.\n+ * Function returns 0 on success and non-zero on failure. A failure to create\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n int ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n@@ -262,6 +264,9 @@ int ref_transaction_create(struct ref_transaction *transaction,\n * Add a reference deletion to transaction. If have_old is true, then\n * old_sha1 holds the value that the reference should have had before\n * the update (which must not be the null SHA-1).\n+ * Function returns 0 on success and non-zero on failure. A failure to delete\n+ * means that the transaction as a whole has failed and will need to be\n+ * rolled back.\n */\n int ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t const char *refname,\n@@ -272,7 +277,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n * Commit all of the changes that have been queued in transaction, as\n * atomically as possible. Return a nonzero value if there is a\n * problem. If err is non-NULL we will add an error string to it to explain\n- * why the transaction failed.\n+ * why the transaction failed. The string does not end in newline.\n */\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err);\n"},{"id":"241756","messageId":"20140515181035.GB26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-2-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 01/44] refs.c: constify the sha arguments for ref_transaction_create|delete|update","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T18:10:36Z","receivedAt":"2014-05-15T18:10:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> ref_transaction_create|delete|update has no need to modify the sha1\n> arguments passed to it so it should use const unsigned char* instead\n> of unsigned char*.\n\nObviously good.\n\nReviewed-by: Jonathan Nieder \n"},{"id":"241757","messageId":"20140515181512.GC26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-3-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T18:15:12Z","receivedAt":"2014-05-15T18:15:12Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Allow ref_transaction_free to be called with NULL and as a result allow\n> ref_transaction_rollback to be called for a NULL transaction.\n>\n> This allows us to write code that will\n> if ( (!transaction ||\n> ref_transaction_update(...)) ||\n> (ref_transaction_commit(...) && !(transaction = NULL)) {\n> ref_transaction_rollback(transaction);\n> ...\n> }\n>\n> In this case transaction is reset to NULL IFF ref_transaction_commit() was\n> invoked and thus the rollback becomes ref_transaction_rollback(NULL) which\n> is safe. IF the conditional triggered prior to ref_transaction_commit()\n> then transaction is untouched and then ref_transaction_rollback(transaction)\n> will rollback the failed transaction.\n\nI still think these last two paragraphs confuse more than enlighten\nhere. There's plenty of time to explain them in the patch that uses\nthat code.\n\nI'd just say something like\n\n\tAllow ref_transaction_free(NULL) and hence ref_transaction_rollback(NULL)\n\tas no-ops.\n\n\tThis makes ref_transaction_rollback easier to use and more similar to\n\tplain 'free'.\n\nAnd maybe:\n\n\tIn particular, it lets us rollback unconditionally as part of cleanup\n\tcode after setting 'transaction = NULL' if a transaction has been\n\tcommitted or rolled back already.\n\nThanks,\nJonathan\n"},{"id":"241758","messageId":"CAL=YDW=5BjCkZ9A-En+mG9CcshC0tCP4j4ZWKSWK5NEfykWy8Q@mail.gmail.com","threadId":"36669","inReplyTo":"20140515181512.GC26471@google.com","subject":"Re: [PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T18:26:10Z","receivedAt":"2014-05-15T18:26:10Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Thanks.\n\nI used your improved text in the commit message.\n\nOn Thu, May 15, 2014 at 11:15 AM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Allow ref_transaction_free to be called with NULL and as a result allow\n>> ref_transaction_rollback to be called for a NULL transaction.\n>>\n>> This allows us to write code that will\n>> if ( (!transaction ||\n>> ref_transaction_update(...)) ||\n>> (ref_transaction_commit(...) && !(transaction = NULL)) {\n>> ref_transaction_rollback(transaction);\n>> ...\n>> }\n>>\n>> In this case transaction is reset to NULL IFF ref_transaction_commit() was\n>> invoked and thus the rollback becomes ref_transaction_rollback(NULL) which\n>> is safe. IF the conditional triggered prior to ref_transaction_commit()\n>> then transaction is untouched and then ref_transaction_rollback(transaction)\n>> will rollback the failed transaction.\n>\n> I still think these last two paragraphs confuse more than enlighten\n> here. There's plenty of time to explain them in the patch that uses\n> that code.\n>\n> I'd just say something like\n>\n> Allow ref_transaction_free(NULL) and hence ref_transaction_rollback(NULL)\n> as no-ops.\n>\n> This makes ref_transaction_rollback easier to use and more similar to\n> plain 'free'.\n>\n> And maybe:\n>\n> In particular, it lets us rollback unconditionally as part of cleanup\n> code after setting 'transaction = NULL' if a transaction has been\n> committed or rolled back already.\n>\n> Thanks,\n> Jonathan\n"},{"id":"241760","messageId":"20140515183836.GD26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-5-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 04/44] refs.c: add an err argument to repack_without_refs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T18:38:36Z","receivedAt":"2014-05-15T18:38:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n> \treturn 0;\n> }\n> \n> -static int repack_without_refs(const char **refnames, int n)\n> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n\nShould this also get an onerr flag to suppress the message to stderr,\nor unconditionally suppress the message to stderr when err != NULL?\n\n[...]\n> @@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)\n> \n> \tif (lock_packed_refs(0)) {\n> \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n> +\t\tif (err)\n> +\t\t\tstrbuf_addf(err, \"cannot delete '%s' from packed refs\",\n> +\t\t\t\t refnames[i]);\n\nunable_to_lock_error is able to come up with a message with more\ndetail (path so the sysadmin can hunt down the problem even if this\nwas run e.g. from a cronjob where the path is not obvious, errno\nhinting at the nature of the problem).\n\n[...]\n> @@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)\n> \t}\n> \n> \t/* Write what remains */\n> -\treturn commit_packed_refs();\n> +\tret = commit_packed_refs();\n> +\tif (ret && err)\n> +\t\tstrbuf_addf(err, \"unable to overwrite old ref-pack file\");\n\nAfter commit_lock_file sets errno, amazingly no one clobbers it\nuntil we get to this point. The only calls in between are to\nfree().\n\nIt would be nice to make that more explicit in commit_packed_refs:\n\n\tint save_errno;\n\n\t...\n\tif (commit_lock_file(packed_ref_cache->lock)) {\n\t\tsave_errno = errno;\n\t\terror = -1;\n\t}\n\n\tpacked_ref_cache->lock = NULL;\n\trelease_packed_ref_cache(packed_ref_cache);\n\n\terrno = save_errno;\n\treturn error;\n\nEven without that, this message could include strerror(errno).\n\n> +\treturn ret;\n> }\n\nThanks,\nJonathan\n"},{"id":"241763","messageId":"xmqqvbt6euym.fsf@gitster.dls.corp.google.com","threadId":"36669","inReplyTo":"20140515180618.GA26471@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-15T18:51:45Z","receivedAt":"2014-05-15T18:51:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Ronnie Sahlberg wrote:\n>\n>> This patch series is based on next and expands on the transaction API.\n>\n> Thanks. Will pick up in v8 where I left off with v6.\n>\n> Applies with just one minor conflict on top of a merge of\n> mh/ref-transaction, rs/ref-update-check-errors-early, and\n> rs/reflog-exists. Here's an interdiff against version 6 for those\n> following along.\n\nThe interdiffs look mostly sensible. Thanks.\n\n>> Version 8:\n>> - Updates after review by JN\n>> - Improve commit messages\n>> - Add a patch that adds an err argument to repack_without_refs\n>> - Add a patch that adds an err argument to delete_loose_ref\n>> - Better document that a _update/_delete/_create failure means the whole\n>> transaction has failed and needs rollback.\n>>\n>> Version 7:\n>> - Updated commit messages per JNs review comments.\n>> - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not\n>> exposed through refs.h\n>\n> diff --git a/builtin/commit.c b/builtin/commit.c\n> index 0f4e1fc..07ccc97 100644\n> --- a/builtin/commit.c\n> +++ b/builtin/commit.c\n> @@ -1684,7 +1684,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n> \t\t\t\t 0, !!current_head, sb.buf) ||\n> \t ref_transaction_commit(transaction, &err)) {\n> \t\trollback_index_files();\n> -\t\tdie(_(\"HEAD: cannot update ref: %s\"), err.buf);\n> +\t\tdie(\"%s\", err.buf);\n> \t}\n> \tref_transaction_free(transaction);\n> \n> diff --git a/builtin/replace.c b/builtin/replace.c\n> index 47c360c..952b589 100644\n> --- a/builtin/replace.c\n> +++ b/builtin/replace.c\n> @@ -163,7 +163,7 @@ static int replace_object(const char *object_ref, const char *replace_ref,\n> \t ref_transaction_update(transaction, ref, repl, prev,\n> \t\t\t\t 0, !is_null_sha1(prev), NULL) ||\n> \t ref_transaction_commit(transaction, &err))\n> -\t\tdie(_(\"%s: failed to replace ref: %s\"), ref, err.buf);\n> +\t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n> \n> \tref_transaction_free(transaction);\n> \treturn 0;\n> diff --git a/builtin/update-ref.c b/builtin/update-ref.c\n> index c5aff92..bd7e96f 100644\n> --- a/builtin/update-ref.c\n> +++ b/builtin/update-ref.c\n> @@ -228,7 +228,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)\n> \n> \tif (ref_transaction_create(transaction, refname, new_sha1,\n> \t\t\t\t update_flags, msg))\n> -\t\tdie(\"failed transaction create for %s\", refname);\n> +\t\tdie(\"cannot create ref '%s'\", refname);\n> \n> \tupdate_flags = 0;\n> \tfree(refname);\n> diff --git a/refs.c b/refs.c\n> index 302a2b3..ed93b75 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -29,6 +29,15 @@ static inline int bad_ref_char(int ch)\n> \treturn 0;\n> }\n> \n> +/** Used as a flag to ref_transaction_delete when a loose ref is beeing\n> + * pruned.\n> + */\n> +#define REF_ISPRUNING\t0x0100\n> +/** Deletion of a ref that only exists as a packed ref in which case we do not\n> + * need to lock the loose ref during the transaction.\n> + */\n> +#define REF_ISPACKONLY\t0x0200\n> +\n> /*\n> * Try to read one refname component from the front of refname. Return\n> * the length of the component found, or -1 if the component is not\n> @@ -2447,12 +2456,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n> \treturn 0;\n> }\n> \n> -static int repack_without_refs(const char **refnames, int n)\n> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n> {\n> \tstruct ref_dir *packed;\n> \tstruct string_list refs_to_delete = STRING_LIST_INIT_DUP;\n> \tstruct string_list_item *ref_to_delete;\n> -\tint i, removed = 0;\n> +\tint i, ret, removed = 0;\n> \n> \t/* Look for a packed ref */\n> \tfor (i = 0; i < n; i++)\n> @@ -2465,6 +2474,9 @@ static int repack_without_refs(const char **refnames, int n)\n> \n> \tif (lock_packed_refs(0)) {\n> \t\tunable_to_lock_error(git_path(\"packed-refs\"), errno);\n> +\t\tif (err)\n> +\t\t\tstrbuf_addf(err, \"cannot delete '%s' from packed refs\",\n> +\t\t\t\t refnames[i]);\n> \t\treturn error(\"cannot delete '%s' from packed refs\", refnames[i]);\n> \t}\n> \tpacked = get_packed_refs(&ref_cache);\n> @@ -2490,20 +2502,28 @@ static int repack_without_refs(const char **refnames, int n)\n> \t}\n> \n> \t/* Write what remains */\n> -\treturn commit_packed_refs();\n> +\tret = commit_packed_refs();\n> +\tif (ret && err)\n> +\t\tstrbuf_addf(err, \"unable to overwrite old ref-pack file\");\n> +\treturn ret;\n> }\n> \n> -static int delete_ref_loose(struct ref_lock *lock, int flag)\n> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n> {\n> \tif (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {\n> \t\t/* loose */\n> -\t\tint err, i = strlen(lock->lk->filename) - 5; /* .lock */\n> +\t\tint res, i = strlen(lock->lk->filename) - 5; /* .lock */\n> \n> \t\tlock->lk->filename[i] = 0;\n> -\t\terr = unlink_or_warn(lock->lk->filename);\n> +\t\tres = unlink_or_warn(lock->lk->filename);\n> \t\tlock->lk->filename[i] = '.';\n> -\t\tif (err && errno != ENOENT)\n> +\t\tif (res && errno != ENOENT) {\n> +\t\t\tif (err)\n> +\t\t\t\tstrbuf_addf(err,\n> +\t\t\t\t\t \"failed to delete loose ref '%s'\",\n> +\t\t\t\t\t lock->lk->filename);\n> \t\t\treturn 1;\n> +\t\t}\n> \t}\n> \treturn 0;\n> }\n> @@ -3483,7 +3503,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t\t\t\t delnames, delnum);\n> \t\tif (!update->lock) {\n> \t\t\tif (err)\n> -\t\t\t\tstrbuf_addf(err ,\"Cannot lock the ref '%s'.\",\n> +\t\t\t\tstrbuf_addf(err, \"Cannot lock the ref '%s'.\",\n> \t\t\t\t\t update->refname);\n> \t\t\tret = 1;\n> \t\t\tgoto cleanup;\n> @@ -3516,13 +3536,14 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\tcontinue;\n> \n> \t\tif (update->lock) {\n> -\t\t\tret |= delete_ref_loose(update->lock, update->type);\n> +\t\t\tret |= delete_ref_loose(update->lock, update->type,\n> +\t\t\t\t\t\terr);\n> \t\t\tif (!(update->flags & REF_ISPRUNING))\n> \t\t\t\tdelnames[delnum++] = update->lock->ref_name;\n> \t\t}\n> \t}\n> \n> -\tret |= repack_without_refs(delnames, delnum);\n> +\tret |= repack_without_refs(delnames, delnum, err);\n> \tfor (i = 0; i < delnum; i++)\n> \t\tunlink_or_warn(git_path(\"logs/%s\", delnames[i]));\n> \tclear_loose_ref_cache(&ref_cache);\n> diff --git a/refs.h b/refs.h\n> index 03b4a7e..0e6d416 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -134,10 +134,6 @@ extern int peel_ref(const char *refname, unsigned char *sha1);\n> \n> /** Locks any ref (for 'HEAD' type refs). */\n> #define REF_NODEREF\t0x01\n> -/** Deleting a loose ref during prune */\n> -#define REF_ISPRUNING\t0x02\n> -/** Deletion of a ref that only exists as a packed ref */\n> -#define REF_ISPACKONLY\t0x04\n> extern struct ref_lock *lock_any_ref_for_update(const char *refname,\n> \t\t\t\t\t\tconst unsigned char *old_sha1,\n> \t\t\t\t\t\tint flags, int *type_p);\n> @@ -240,6 +236,9 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n> * be deleted. If have_old is true, then old_sha1 holds the value\n> * that the reference should have had before the update, or zeros if\n> * it must not have existed beforehand.\n> + * Function returns 0 on success and non-zero on failure. A failure to update\n> + * means that the transaction as a whole has failed and will need to be\n> + * rolled back.\n> */\n> int ref_transaction_update(struct ref_transaction *transaction,\n> \t\t\t const char *refname,\n> @@ -252,6 +251,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n> * that the reference should have after the update; it must not be the\n> * null SHA-1. It is verified that the reference does not exist\n> * already.\n> + * Function returns 0 on success and non-zero on failure. A failure to create\n> + * means that the transaction as a whole has failed and will need to be\n> + * rolled back.\n> */\n> int ref_transaction_create(struct ref_transaction *transaction,\n> \t\t\t const char *refname,\n> @@ -262,6 +264,9 @@ int ref_transaction_create(struct ref_transaction *transaction,\n> * Add a reference deletion to transaction. If have_old is true, then\n> * old_sha1 holds the value that the reference should have had before\n> * the update (which must not be the null SHA-1).\n> + * Function returns 0 on success and non-zero on failure. A failure to delete\n> + * means that the transaction as a whole has failed and will need to be\n> + * rolled back.\n> */\n> int ref_transaction_delete(struct ref_transaction *transaction,\n> \t\t\t const char *refname,\n> @@ -272,7 +277,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n> * Commit all of the changes that have been queued in transaction, as\n> * atomically as possible. Return a nonzero value if there is a\n> * problem. If err is non-NULL we will add an error string to it to explain\n> - * why the transaction failed.\n> + * why the transaction failed. The string does not end in newline.\n> */\n> int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t struct strbuf *err);\n"},{"id":"241765","messageId":"20140515190434.GF26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-7-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T19:04:34Z","receivedAt":"2014-05-15T19:04:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> [Subject: refs.c: add an err argument ro delete_loose_ref]\n\ns/ro/to/\ns/delete_loose_ref/delete_ref_loose/\n\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)\n> \treturn repack_without_refs(&refname, 1, NULL);\n> }\n> \n> -static int delete_ref_loose(struct ref_lock *lock, int flag)\n> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n\nShould this get an onerr flag to suppress the message to stderr\nor unconditionally suppress it when err != NULL?\n\n[...]\n> \t\tlock->lk->filename[i] = 0;\n> -\t\terr = unlink_or_warn(lock->lk->filename);\n> +\t\tres = unlink_or_warn(lock->lk->filename);\n\nIt seems like in the new error handling scheme there should be a new\nvariant on wrapper.c's warn_if_unremovable:\n\n\tstatic int add_err_if_unremovable(const char *op, const char *file, struct strbuf *err, int rc)\n\t{\n\t\tint err = errno;\n\t\tif (rc < 0 && err != ENOENT) {\n\t\t\tstrbuf_addf(err, \"unable to %s %s: %s\",\n\t\t\t\t op, file, strerror(errno));\n\t\t\terrno = err;\n\t\t}\n\t\treturn rc;\n\t}\n\n\tstatic int unlink_or_err(const char *file, struct strbuf *err)\n\t{\n\t\treturn add_err_if_unremovable(\"unlink\", file, err, unlink(file));\n\t}\n\n\t\tres = unlink_or_err(lock->lk->filename, err);\n"},{"id":"241771","messageId":"20140515192315.GG26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-9-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 08/44] update-ref.c: log transaction error from the update_ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T19:23:15Z","receivedAt":"2014-05-15T19:23:15Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Call ref_transaction_commit with QUIET_ON_ERR and use the strbuf that is\n> returned to print a log message if/after the transaction fails.\n>\n> Signed-off-by: Ronnie Sahlberg \n\nAll error paths in _commit add to the error string now, so the only\neffect should be to add the missing error message when commiting\npacked-refs fails (thanks for fixing that) and to change some one-line\nerrors to two-line.\n\nReviewed-by: Jonathan Nieder \n"},{"id":"241773","messageId":"20140515193446.GH26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-11-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T19:34:46Z","receivedAt":"2014-05-15T19:34:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Update ref_transaction_update() do some basic error checking and return\n> non-zero on error. Update all callers to check ref_transaction_update() for\n> error. There are currently no conditions in _update that will return error but\n> there will be in the future.\n\nProbably worth passing a 'struct strbuf *err' argument. Then callers\ncan do\n\n\t\tdie(\"%s\", err.buf);\n\nand the error message can say which ref and whether we were trying to\ncreate a ref, or delete one, or whatever.\n\n> --- a/builtin/update-ref.c\n> +++ b/builtin/update-ref.c\n> @@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)\n> \tif (*next != line_termination)\n> \t\tdie(\"update %s: extra input: %s\", refname, next);\n> \n> -\tref_transaction_update(transaction, refname, new_sha1, old_sha1,\n> -\t\t\t update_flags, have_old);\n> +\tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n> +\t\t\t\t update_flags, have_old))\n> +\t\tdie(\"update %s: failed\", refname);\n\nThis could say\n\n\t\tdie(\"update %s: %s\", refname, err.buf);\n\nto give context about which command it was trying to execute.\n\n[...]\n> @@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)\n> \tif (*next != line_termination)\n> \t\tdie(\"verify %s: extra input: %s\", refname, next);\n> \n> -\tref_transaction_update(transaction, refname, new_sha1, old_sha1,\n> -\t\t\t update_flags, have_old);\n> +\tif (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n> +\t\t\t\t update_flags, have_old))\n> +\t\tdie(\"failed transaction update for %s\", refname);\n\nAnd this could say\n\n\t\tdie(\"verify %s: %s\", refname, err.buf);\n\n[...]\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n> * be deleted. If have_old is true, then old_sha1 holds the value\n> * that the reference should have had before the update, or zeros if\n> * it must not have existed beforehand.\n> + * Function returns 0 on success and non-zero on failure. A failure to update\n> + * means that the transaction as a whole has failed and will need to be\n> + * rolled back.\n> + */\n\nThanks for this documentation.\n"},{"id":"241776","messageId":"20140515194456.GI26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-12-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T19:44:56Z","receivedAt":"2014-05-15T19:44:56Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Do basic error checking in ref_transaction_create() and make it return\n> non-zero on error.\n\nSame thoughts as _update(). Basic idea is good but would be nice to\nhave a 'struct strbuf *err' parameter.\n\nThanks,\nJonathan\n"},{"id":"241778","messageId":"20140515195152.GJ26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-13-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T19:51:52Z","receivedAt":"2014-05-15T19:51:52Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change ref_transaction_delete() to do basic error checking and return\n> non-zero of error.\n\nLikewise: a 'struct strbuf *err' would make nicer error messages\npossible.\n"},{"id":"241781","messageId":"CAL=YDWkE4WnE1ELRJcXFEcNhSw=gv9XjqvrK_EqD3PQu8i0URA@mail.gmail.com","threadId":"36669","inReplyTo":"20140515190434.GF26471@google.com","subject":"Re: [PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T20:00:10Z","receivedAt":"2014-05-15T20:00:10Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 12:04 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> [Subject: refs.c: add an err argument ro delete_loose_ref]\n>\n> s/ro/to/\n> s/delete_loose_ref/delete_ref_loose/\n>\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)\n>> return repack_without_refs(&refname, 1, NULL);\n>> }\n>>\n>> -static int delete_ref_loose(struct ref_lock *lock, int flag)\n>> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n>\n> Should this get an onerr flag to suppress the message to stderr\n> or unconditionally suppress it when err != NULL?\n>\n\nFixed.\nI added a new function unlink_or_err that will update err if non-NULL\nand unse warning() otherwise.\n\n\n> [...]\n>> lock->lk->filename[i] = 0;\n>> - err = unlink_or_warn(lock->lk->filename);\n>> + res = unlink_or_warn(lock->lk->filename);\n>\n> It seems like in the new error handling scheme there should be a new\n> variant on wrapper.c's warn_if_unremovable:\n>\n> static int add_err_if_unremovable(const char *op, const char *file, struct strbuf *err, int rc)\n> {\n> int err = errno;\n> if (rc < 0 && err != ENOENT) {\n> strbuf_addf(err, \"unable to %s %s: %s\",\n> op, file, strerror(errno));\n> errno = err;\n> }\n> return rc;\n> }\n>\n> static int unlink_or_err(const char *file, struct strbuf *err)\n> {\n> return add_err_if_unremovable(\"unlink\", file, err, unlink(file));\n> }\n>\n> res = unlink_or_err(lock->lk->filename, err);\n"},{"id":"241783","messageId":"20140515211156.GK26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-14-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 13/44] tag.c: use ref transactions when doing updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T21:11:56Z","receivedAt":"2014-05-15T21:11:56Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/builtin/tag.c\n> +++ b/builtin/tag.c\n> @@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n> \tif (annotate)\n> \t\tcreate_tag(object, tag, &buf, &opt, prev, object);\n> \n> -\tlock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);\n> -\tif (!lock)\n> -\t\tdie(_(\"%s: cannot lock the ref\"), ref.buf);\n> -\tif (write_ref_sha1(lock, object, NULL) < 0)\n> -\t\tdie(_(\"%s: cannot update the ref\"), ref.buf);\n> +\ttransaction = ref_transaction_begin();\n> +\tif (!transaction ||\n> +\t ref_transaction_update(transaction, ref.buf, object, prev,\n> +\t\t\t\t 0, !is_null_sha1(prev)) ||\n> +\t ref_transaction_commit(transaction, NULL, &err))\n> +\t\tdie(_(\"%s: cannot update the ref: %s\"), ref.buf, err.buf);\n\nThe error string says what ref it was trying to update, so\n\n\t\tdie(\"%s\", err.buf);\n\nshould be enough. (E.g.,\n\n\tfatal: refs/tags/v1.0: cannot lock the ref\n\nwould become\n\n\tfatal: Cannot lock the ref 'refs/tags/v1.0'.\n\ninstead of\n\n\tfatal: refs/tags/v1.0: cannot update the ref: Cannot lock the ref 'refs/tags/v1.0'.\n\n.)\n\nThanks,\nJonathan\n"},{"id":"241784","messageId":"20140515211835.GL26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-15-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 14/44] replace.c: use the ref transaction functions for updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T21:18:35Z","receivedAt":"2014-05-15T21:18:35Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/builtin/replace.c\n> +++ b/builtin/replace.c\n[...]\n> @@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,\n> \telse if (!force)\n> \t\tdie(\"replace ref '%s' already exists\", ref);\n> \n> -\tlock = lock_any_ref_for_update(ref, prev, 0, NULL);\n> -\tif (!lock)\n> -\t\tdie(\"%s: cannot lock the ref\", ref);\n> -\tif (write_ref_sha1(lock, repl, NULL) < 0)\n> -\t\tdie(\"%s: cannot update the ref\", ref);\n> +\ttransaction = ref_transaction_begin();\n> +\tif (!transaction ||\n> +\t ref_transaction_update(transaction, ref, repl, prev,\n> +\t\t\t\t 0, !is_null_sha1(prev)) ||\n> +\t ref_transaction_commit(transaction, NULL, &err))\n> +\t\tdie(\"%s: failed to replace ref: %s\", ref, err.buf);\n\nThis would write\n\n\tfatal: refs/replace/09c779943364d893c190066c385e6112af421db3: failed to replace ref: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n\nPerhaps something like\n\n\t$ git replace foo bar\n\tfatal: replace foo: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n\nwould make sense (die(\"replace %s: %s\", object_ref, err.buf)). Plain\n\n\t$ git replace foo bar\n\tfatal: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n\nalso seems fine (die(\"%s\", err.buf)).\n\nThanks,\nJonathan\n"},{"id":"241785","messageId":"20140515212130.GM26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-16-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 15/44] commit.c: use ref transactions for updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T21:21:30Z","receivedAt":"2014-05-15T21:21:30Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change commit.c to use ref transactions for all ref updates.\n> Make sure we pass a NULL pointer to ref_transaction_update if have_old\n> is false.\n\nOnce ref_transaction_begin() and ref_transaction_update() set err,\nthis will always have a reasonable message to print in die().\n\nReviewed-by: Jonathan Nieder \n"},{"id":"241787","messageId":"20140515214745.GN26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-18-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T21:47:45Z","receivedAt":"2014-05-15T21:47:45Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -1679,39 +1679,45 @@ found_entry:\n> static int update_branch(struct branch *b)\n> {\n> \tstatic const char *msg = \"fast-import\";\n> -\tstruct ref_lock *lock;\n> +\tstruct ref_transaction *transaction;\n> \tunsigned char old_sha1[20];\n> +\tstruct strbuf err = STRBUF_INIT;\n> \n> \tif (read_ref(b->name, old_sha1))\n> \t\thashclr(old_sha1);\n> +\n> \tif (is_null_sha1(b->sha1)) {\n> \t\tif (b->delete)\n> \t\t\tdelete_ref(b->name, old_sha1, 0);\n> \t\treturn 0;\n> \t}\n> -\tlock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);\n> -\tif (!lock)\n> -\t\treturn error(\"Unable to lock %s\", b->name);\n> \tif (!force_update && !is_null_sha1(old_sha1)) {\n> \t\tstruct commit *old_cmit, *new_cmit;\n> \n> \t\told_cmit = lookup_commit_reference_gently(old_sha1, 0);\n> \t\tnew_cmit = lookup_commit_reference_gently(b->sha1, 0);\n> \t\tif (!old_cmit || !new_cmit) {\n> -\t\t\tunlock_ref(lock);\n> \t\t\treturn error(\"Branch %s is missing commits.\", b->name);\n> \t\t}\n\n(style) Now that there's only one line in the \"if\" body, we can\ndrop the braces.\n\n> \n> \t\tif (!in_merge_bases(old_cmit, new_cmit)) {\n> -\t\t\tunlock_ref(lock);\n> \t\t\twarning(\"Not updating %s\"\n> \t\t\t\t\" (new tip %s does not contain %s)\",\n> \t\t\t\tb->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));\n> \t\t\treturn -1;\n\n(not about this patch, feel free to ignore) This could\nreturn warning(\"...\")\n\n> \t}\n> -\tif (write_ref_sha1(lock, b->sha1, msg) < 0)\n> -\t\treturn error(\"Unable to update %s\", b->name);\n> +\ttransaction = ref_transaction_begin();\n> +\tif ((!transaction ||\n> +\t ref_transaction_update(transaction, b->name, b->sha1, old_sha1,\n> +\t\t\t\t 0, 1)) ||\n\nWould be more idiomatic to drop a layer of parentheses:\n\n\tif (!transaction ||\n\t ref_transaction_update(...) ||\n\n> +\t (ref_transaction_commit(transaction, msg, &err) &&\n> +\t !(transaction = NULL))) {\n\nWould be clearer if ref_transaction_commit didn't take care of the\nrollback (or in other words if patch 21 were earlier in the series).\n\n> +\t\tref_transaction_rollback(transaction);\n> +\t\terror(\"Unable to update branch %s: %s\", b->name, err.buf);\n> +\t\tstrbuf_release(&err);\n> +\t\treturn -1;\n> +\t}\n\nExample old message:\n\n\terror: Unable to lock refs/heads/master\n\nNew message:\n\n\terror: Unable to update branch refs/heads/master: Cannot lock the ref 'refs/heads/master'.\n\nSo 'error(\"%s\", err.buf)' would probably work better.\n\nThe only call site is dump_branches:\n\n\tfor (i = 0; i < branch_table_sz; i++) {\n\t\tfor (b = branch_table[i]; b; b = b->table_next_branch)\n\t\t\tfailure |= update_branch(b);\n\t}\n\nShould these happen in a single transaction? I haven't thought\nthrough whether it would be a good idea, if it should be optional, or\nwhat.\n\nAnyway, that would be a bigger behavior change, but it's interesting\nto think about.\n\nThanks,\nJonathan\n"},{"id":"241788","messageId":"20140515215307.GO26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-17-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 16/44] sequencer.c: use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T21:53:07Z","receivedAt":"2014-05-15T21:53:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -272,23 +272,31 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,\n[...]\n> +\t\terror(_(\"HEAD: Could not fast-forward: %s\\n\"), err.buf);\n\nAs with 'git commit', since err.buf already mentions that HEAD was the\nproblematic ref I think it has enough information for the user to\ndiagnose the problem and figure out what to do next.\n\n\t\terror(\"%s\", err.buf);\n"},{"id":"241789","messageId":"CAL=YDWn8uFOAfyySRcdnVjS4KSifp2QS4vM6GOX_UAx4yz7VXA@mail.gmail.com","threadId":"36669","inReplyTo":"20140515195152.GJ26471@google.com","subject":"Re: [PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:01:39Z","receivedAt":"2014-05-15T22:01:39Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 12:51 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Change ref_transaction_delete() to do basic error checking and return\n>> non-zero of error.\n>\n> Likewise: a 'struct strbuf *err' would make nicer error messages\n> possible.\n\nDone.\n"},{"id":"241790","messageId":"CAL=YDWnDFxJjHTXzFu9bGVQRTohs4nP-49eXObOQEcEykzmmJw@mail.gmail.com","threadId":"36669","inReplyTo":"20140515194456.GI26471@google.com","subject":"Re: [PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:02:39Z","receivedAt":"2014-05-15T22:02:39Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 12:44 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Do basic error checking in ref_transaction_create() and make it return\n>> non-zero on error.\n>\n> Same thoughts as _update(). Basic idea is good but would be nice to\n> have a 'struct strbuf *err' parameter.\n\nDone.\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"241791","messageId":"CAL=YDWmk6vR0J0fs6jDPgEaAXLVk5xGbR-wPToyccxuSe-MKuw@mail.gmail.com","threadId":"36669","inReplyTo":"20140515193446.GH26471@google.com","subject":"Re: [PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:09:14Z","receivedAt":"2014-05-15T22:09:14Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 12:34 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Update ref_transaction_update() do some basic error checking and return\n>> non-zero on error. Update all callers to check ref_transaction_update() for\n>> error. There are currently no conditions in _update that will return error but\n>> there will be in the future.\n>\n> Probably worth passing a 'struct strbuf *err' argument. Then callers\n> can do\n>\n> die(\"%s\", err.buf);\n>\n\nDone.\n\n> and the error message can say which ref and whether we were trying to\n> create a ref, or delete one, or whatever.\n>\n>> --- a/builtin/update-ref.c\n>> +++ b/builtin/update-ref.c\n>> @@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)\n>> if (*next != line_termination)\n>> die(\"update %s: extra input: %s\", refname, next);\n>>\n>> - ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n>> - update_flags, have_old);\n>> + if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n>> + update_flags, have_old))\n>> + die(\"update %s: failed\", refname);\n>\n> This could say\n>\n> die(\"update %s: %s\", refname, err.buf);\n\nDone.\n\n>\n> to give context about which command it was trying to execute.\n>\n> [...]\n>> @@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)\n>> if (*next != line_termination)\n>> die(\"verify %s: extra input: %s\", refname, next);\n>>\n>> - ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n>> - update_flags, have_old);\n>> + if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,\n>> + update_flags, have_old))\n>> + die(\"failed transaction update for %s\", refname);\n>\n> And this could say\n>\n> die(\"verify %s: %s\", refname, err.buf);\n\nDone.\n\n>\n> [...]\n>> --- a/refs.h\n>> +++ b/refs.h\n>> @@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);\n>> * be deleted. If have_old is true, then old_sha1 holds the value\n>> * that the reference should have had before the update, or zeros if\n>> * it must not have existed beforehand.\n>> + * Function returns 0 on success and non-zero on failure. A failure to update\n>> + * means that the transaction as a whole has failed and will need to be\n>> + * rolled back.\n>> + */\n>\n> Thanks for this documentation.\n"},{"id":"241793","messageId":"CAL=YDWm-9Bq+bpqEieUuARzGO6+3ubbsLPg5cU+2g3OiYTdm8g@mail.gmail.com","threadId":"36669","inReplyTo":"20140515214745.GN26471@google.com","subject":"Re: [PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:20:32Z","receivedAt":"2014-05-15T22:20:32Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 2:47 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/fast-import.c\n>> +++ b/fast-import.c\n>> @@ -1679,39 +1679,45 @@ found_entry:\n>> static int update_branch(struct branch *b)\n>> {\n>> static const char *msg = \"fast-import\";\n>> - struct ref_lock *lock;\n>> + struct ref_transaction *transaction;\n>> unsigned char old_sha1[20];\n>> + struct strbuf err = STRBUF_INIT;\n>>\n>> if (read_ref(b->name, old_sha1))\n>> hashclr(old_sha1);\n>> +\n>> if (is_null_sha1(b->sha1)) {\n>> if (b->delete)\n>> delete_ref(b->name, old_sha1, 0);\n>> return 0;\n>> }\n>> - lock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);\n>> - if (!lock)\n>> - return error(\"Unable to lock %s\", b->name);\n>> if (!force_update && !is_null_sha1(old_sha1)) {\n>> struct commit *old_cmit, *new_cmit;\n>>\n>> old_cmit = lookup_commit_reference_gently(old_sha1, 0);\n>> new_cmit = lookup_commit_reference_gently(b->sha1, 0);\n>> if (!old_cmit || !new_cmit) {\n>> - unlock_ref(lock);\n>> return error(\"Branch %s is missing commits.\", b->name);\n>> }\n>\n> (style) Now that there's only one line in the \"if\" body, we can\n> drop the braces.\n>\n\nDone.\n\n>>\n>> if (!in_merge_bases(old_cmit, new_cmit)) {\n>> - unlock_ref(lock);\n>> warning(\"Not updating %s\"\n>> \" (new tip %s does not contain %s)\",\n>> b->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));\n>> return -1;\n>\n> (not about this patch, feel free to ignore) This could\n> return warning(\"...\")\n\nIgnored.\n\n>\n>> }\n>> - if (write_ref_sha1(lock, b->sha1, msg) < 0)\n>> - return error(\"Unable to update %s\", b->name);\n>> + transaction = ref_transaction_begin();\n>> + if ((!transaction ||\n>> + ref_transaction_update(transaction, b->name, b->sha1, old_sha1,\n>> + 0, 1)) ||\n>\n> Would be more idiomatic to drop a layer of parentheses:\n>\n> if (!transaction ||\n> ref_transaction_update(...) ||\n>\n>> + (ref_transaction_commit(transaction, msg, &err) &&\n>> + !(transaction = NULL))) {\n>\n\nDone.\n\n\n> Would be clearer if ref_transaction_commit didn't take care of the\n> rollback (or in other words if patch 21 were earlier in the series).\n\nTrue, but first making it correct, but ugly, here makes one appreciate\npatch 21 more. That patch makes it correct and not ugly :-)\nLeaving as is.\n\n>\n>> + ref_transaction_rollback(transaction);\n>> + error(\"Unable to update branch %s: %s\", b->name, err.buf);\n>> + strbuf_release(&err);\n>> + return -1;\n>> + }\n>\n> Example old message:\n>\n> error: Unable to lock refs/heads/master\n>\n> New message:\n>\n> error: Unable to update branch refs/heads/master: Cannot lock the ref 'refs/heads/master'.\n>\n> So 'error(\"%s\", err.buf)' would probably work better.\n\nDone.\n\n>\n> The only call site is dump_branches:\n>\n> for (i = 0; i < branch_table_sz; i++) {\n> for (b = branch_table[i]; b; b = b->table_next_branch)\n> failure |= update_branch(b);\n> }\n>\n> Should these happen in a single transaction? I haven't thought\n> through whether it would be a good idea, if it should be optional, or\n> what.\n>\n> Anyway, that would be a bigger behavior change, but it's interesting\n> to think about.\n\nI will try to add it to the next patch series.\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"241794","messageId":"CAL=YDWn5hNte5dAyddyHaGoYV3+8gW=qj2KqDkWxn+djQgOcbA@mail.gmail.com","threadId":"36669","inReplyTo":"20140515211156.GK26471@google.com","subject":"Re: [PATCH v8 13/44] tag.c: use ref transactions when doing updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:27:08Z","receivedAt":"2014-05-15T22:27:08Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 2:11 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/builtin/tag.c\n>> +++ b/builtin/tag.c\n>> @@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n>> if (annotate)\n>> create_tag(object, tag, &buf, &opt, prev, object);\n>>\n>> - lock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);\n>> - if (!lock)\n>> - die(_(\"%s: cannot lock the ref\"), ref.buf);\n>> - if (write_ref_sha1(lock, object, NULL) < 0)\n>> - die(_(\"%s: cannot update the ref\"), ref.buf);\n>> + transaction = ref_transaction_begin();\n>> + if (!transaction ||\n>> + ref_transaction_update(transaction, ref.buf, object, prev,\n>> + 0, !is_null_sha1(prev)) ||\n>> + ref_transaction_commit(transaction, NULL, &err))\n>> + die(_(\"%s: cannot update the ref: %s\"), ref.buf, err.buf);\n>\n> The error string says what ref it was trying to update, so\n>\n> die(\"%s\", err.buf);\n>\n> should be enough. (E.g.,\n>\n> fatal: refs/tags/v1.0: cannot lock the ref\n>\n> would become\n>\n> fatal: Cannot lock the ref 'refs/tags/v1.0'.\n>\n> instead of\n>\n> fatal: refs/tags/v1.0: cannot update the ref: Cannot lock the ref 'refs/tags/v1.0'.\n>\n> .)\n>\n\nDone.\n\nThanks!\n"},{"id":"241795","messageId":"CAL=YDWk0S4StNcZhz6JDBEe2dpvVsCTqROZ+100Cv012t9FNrg@mail.gmail.com","threadId":"36669","inReplyTo":"20140515211835.GL26471@google.com","subject":"Re: [PATCH v8 14/44] replace.c: use the ref transaction functions for updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T22:30:39Z","receivedAt":"2014-05-15T22:30:39Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 2:18 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/builtin/replace.c\n>> +++ b/builtin/replace.c\n> [...]\n>> @@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,\n>> else if (!force)\n>> die(\"replace ref '%s' already exists\", ref);\n>>\n>> - lock = lock_any_ref_for_update(ref, prev, 0, NULL);\n>> - if (!lock)\n>> - die(\"%s: cannot lock the ref\", ref);\n>> - if (write_ref_sha1(lock, repl, NULL) < 0)\n>> - die(\"%s: cannot update the ref\", ref);\n>> + transaction = ref_transaction_begin();\n>> + if (!transaction ||\n>> + ref_transaction_update(transaction, ref, repl, prev,\n>> + 0, !is_null_sha1(prev)) ||\n>> + ref_transaction_commit(transaction, NULL, &err))\n>> + die(\"%s: failed to replace ref: %s\", ref, err.buf);\n>\n> This would write\n>\n> fatal: refs/replace/09c779943364d893c190066c385e6112af421db3: failed to replace ref: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n>\n> Perhaps something like\n>\n> $ git replace foo bar\n> fatal: replace foo: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n>\n> would make sense (die(\"replace %s: %s\", object_ref, err.buf)). Plain\n>\n> $ git replace foo bar\n> fatal: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.\n>\n> also seems fine (die(\"%s\", err.buf)).\n\nDone.\n\nI used : die(\"%s\", err.buf)\nsince this is consistent with most of the other if-transaction-failed-die()\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"241800","messageId":"20140515225804.GQ26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-19-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 18/44] branch.c: use ref transaction for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T22:58:04Z","receivedAt":"2014-05-15T22:58:04Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change create_branch to use a ref transaction when creating the new branch.\n> ref_transaction_create will check that the ref does not already exist and fail\n> otherwise meaning that we no longer need to keep a lock on the ref during the\n> setup_tracking. This simplifies the code since we can now do the transaction\n> in one single step.\n\nPreviously the ref lock was also held during setup_tracking, which\nsets up configuration for the branch in .git/config. So in principle\nthey were one transaction and this patch would change that.\n\nThe race:\n\n\tcheckout -B master origin/master\n\t--------------------------------\n\t update-ref -d refs/heads/master\n\t -------------------------------\n\t checkout -B master other/master\n\t -------------------------------\n\tcreate ref\n\t delete ref\n\t create ref\n\n\tconfigure tracking configure tracking\n\nSince setup_tracking is not a single transaction, if the two processes\nare lucky enough to not try to write to .git/config at the same time\nthen the resulting configuration could have branch.master.merge set\nby the first checkout -b and branch.master.remote set by the second.\n\nBut trying to \"checkout -b\" the same branch in two terminals\nconcurrently is a somewhat insane thing to do, so I don't mind\nbreaking it.\n\n[...]\n> This also fixes a race condition in the old code where two concurrent\n> create_branch could race since the lock_any_ref_for_update/write_ref_sha1\n> did not protect against the ref already existsing. I.e. one thread could end up\n\ns/existsing/existing/\n\n> overwriting a branch even if the forcing flag is false.\n\nGood catch.\n\n> --- a/branch.c\n> +++ b/branch.c\n[...]\n> @@ -301,13 +291,25 @@ void create_branch(const char *head,\n[...]\n> +\t\tif (!transaction ||\n> +\t\t\tref_transaction_update(transaction, ref.buf, sha1,\n> +\t\t\t\t\t null_sha1, 0, !forcing) ||\n> +\t\t\tref_transaction_commit(transaction, msg, &err))\n> +\t\t\t\tdie_errno(_(\"%s: failed to write ref: %s\"),\n> +\t\t\t\t\t ref.buf, err.buf);\n\nerrno is not guaranteed valid here. The usual\n\n\t\t\t\tdie(\"%s\", err.buf);\n\nshould do the trick.\n\nWith the changes mentioned above (commit message typofix, die()\nmessage),\nReviewed-by: Jonathan Nieder \n\nThanks.\n"},{"id":"241802","messageId":"CAL=YDWk5h48sS1A61nsBTFCBQ327nZ5mRcr3Jp4f-yA4Gqe4Gg@mail.gmail.com","threadId":"36669","inReplyTo":"20140515183836.GD26471@google.com","subject":"Re: [PATCH v8 04/44] refs.c: add an err argument to repack_without_refs","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-15T23:06:59Z","receivedAt":"2014-05-15T23:06:59Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 11:38 AM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)\n>> return 0;\n>> }\n>>\n>> -static int repack_without_refs(const char **refnames, int n)\n>> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)\n>\n> Should this also get an onerr flag to suppress the message to stderr,\n> or unconditionally suppress the message to stderr when err != NULL?\n\nUnconditionally suppress the message I think. And is what I did.\n\n>\n> [...]\n>> @@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)\n>>\n>> if (lock_packed_refs(0)) {\n>> unable_to_lock_error(git_path(\"packed-refs\"), errno);\n>> + if (err)\n>> + strbuf_addf(err, \"cannot delete '%s' from packed refs\",\n>> + refnames[i]);\n>\n> unable_to_lock_error is able to come up with a message with more\n> detail (path so the sysadmin can hunt down the problem even if this\n> was run e.g. from a cronjob where the path is not obvious, errno\n> hinting at the nature of the problem).\n\nI changed it so that if err is non-NULL we call a new function that updates the\nstrbuf with a nice message explaining the failure and then return 1\nwithout writing anything to stderr.\n\nFor the NULL case I left it to continue logging two lines of errors\n\"Unable to create '%s.lock': %s.\\n\\n\"\n\"cannot delete '%s' from packed refs\"\n\nThat second line should probably go away since the error at this stage\nis related to locking the packed\nrefs file itself and has nothing really to do with a particular ref\nsuch as refnames[i]\n\n\n\n>\n> [...]\n>> @@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)\n>> }\n>>\n>> /* Write what remains */\n>> - return commit_packed_refs();\n>> + ret = commit_packed_refs();\n>> + if (ret && err)\n>> + strbuf_addf(err, \"unable to overwrite old ref-pack file\");\n>\n> After commit_lock_file sets errno, amazingly no one clobbers it\n> until we get to this point. The only calls in between are to\n> free().\n>\n> It would be nice to make that more explicit in commit_packed_refs:\n>\n> int save_errno;\n>\n> ...\n> if (commit_lock_file(packed_ref_cache->lock)) {\n> save_errno = errno;\n> error = -1;\n> }\n>\n> packed_ref_cache->lock = NULL;\n> release_packed_ref_cache(packed_ref_cache);\n>\n> errno = save_errno;\n> return error;\n>\n> Even without that, this message could include strerror(errno).\n\nDone and Done.\n\nThanks.\n"},{"id":"241822","messageId":"20140515231615.GR26471@google.com","threadId":"36669","inReplyTo":"1400174999-26786-20-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 19/44] refs.c: change update_ref to use a transaction","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-15T23:16:15Z","receivedAt":"2014-05-15T23:16:15Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change the update_ref helper function to use a ref transaction internally.\n> \n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 25 +++++++++++++++++++++----\n> 1 file changed, 21 insertions(+), 4 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 6e5e940..0476892 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -3416,11 +3416,28 @@ int update_ref(const char *action, const char *refname,\n> \t const unsigned char *sha1, const unsigned char *oldval,\n> \t int flags, enum action_on_err onerr)\n> {\n> -\tstruct ref_lock *lock;\n> -\tlock = update_ref_lock(refname, oldval, flags, NULL, onerr);\n> -\tif (!lock)\n> +\tstruct ref_transaction *t;\n> +\tstruct strbuf err = STRBUF_INIT;\n> +\n> +\tt = ref_transaction_begin();\n> +\tif ((!t ||\n> +\t ref_transaction_update(t, refname, sha1, oldval, flags,\n> +\t\t\t\t !!oldval)) ||\n> +\t (ref_transaction_commit(t, action, &err) && !(t = NULL))) {\n> +\t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n> +\n> +\t\tref_transaction_rollback(t);\n> +\t\tswitch (onerr) {\n> +\t\tcase UPDATE_REFS_MSG_ON_ERR:\n> +\t\t\terror(str, refname, err.buf); break;\n> +\t\tcase UPDATE_REFS_DIE_ON_ERR:\n> +\t\t\tdie(str, refname, err.buf); break;\n> +\t\tcase UPDATE_REFS_QUIET_ON_ERR: break;\n> +\t\t}\n\nThe error string already contains the refname, so it would make sense to\ndo\n\n\t\tcase UPDATE_REFS_MSG_ON_ERR:\n\t\t\terror(\"%s\", err.buf); break;\n\netc here.\n\nIt occurs to me now that if ref_transaction_begin fails, presumably\nthe error string would not contain the refname. I want to say that\nthe refname wouldn't help much in diagnosing or recovering from the\nproblem in that case so it would still be okay to just do error(\"%s\",\nerr.buf). What do you think?\n\nThanks,\nJonathan\n"},{"id":"241850","messageId":"20140516002029.GB11011@google.com","threadId":"36669","inReplyTo":"1400174999-26786-22-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-16T00:20:29Z","receivedAt":"2014-05-16T00:20:29Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change ref_transaction_commit so that it does not free the transaction.\n> Instead require that a caller will end a transaction by either calling\n> ref_transaction_rollback or ref_transaction_free.\n\nCan I always use ref_transaction_rollback instead of ref_transaction_free?\nIt would be convenient if my cleanup code could always call _rollback\ninstead of having to do something different for success and errors.\n\nAnother way to ask the question: what is it valid to do with a\ntransaction after commiting?\n\nThanks,\nJonathan\n"},{"id":"241924","messageId":"CAL=YDWkdjjMmVNcXHYr2CGsR4-eU0aHHR5Ek6JipL-Xhi=p7-w@mail.gmail.com","threadId":"36669","inReplyTo":"20140516002029.GB11011@google.com","subject":"Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-16T15:02:23Z","receivedAt":"2014-05-16T15:02:23Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 15, 2014 at 5:20 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Change ref_transaction_commit so that it does not free the transaction.\n>> Instead require that a caller will end a transaction by either calling\n>> ref_transaction_rollback or ref_transaction_free.\n>\n> Can I always use ref_transaction_rollback instead of ref_transaction_free?\n> It would be convenient if my cleanup code could always call _rollback\n> instead of having to do something different for success and errors.\n\nCurrently, yes.\n\nBut it might make sense to change these so rollback only clears the\nupdates that are in flight from the transaction and\nfree only frees the transaction itself iff there are no updates in flight.\n\nI.e. the success and error would then differ like this :\n...\n if (transaction_commit()) {\n transaction_rollback()\n transaction_free()\n return error(\"some error\")\n }\n transaction_free()\n\n\n\n>\n> Another way to ask the question: what is it valid to do with a\n> transaction after commiting?\n\nRight now the only valid thing to do is either rollback or free. But\nwe could allow other things too :\n\n\nre-usable transactions.\n---------------------------------\nI don't know if this is a good reason or not, but one reason we might\nwant to keep\ntwo different names could be if we want to start allowing to re-use\ntransactions.\nFor example for cases/backends where transaction_begin() might be very\nexpensive.\n\nFor that case I would imagine we could allow to do things such as\n\nt = transaction_begin()\n...\n/* first transaction */\ntransaction_update(...)\ntransaction_commit(...)\n if transaction failed transaction_rollback(...)\n\n/* second transaction, first transaction cleared all updates in\nflight either through commit or through rollback */\ntransaction_update()\ntransaction_commit()\n if transaction failed transaction_rollback(...)\n...\ntransaction_free()\n\n\n(In order to do something like this we would still need to do some\nchanges so that rollback will only free the updates that were in\nflight but not free the transaction itself.)\n\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"241925","messageId":"CAL=YDW=xfdBvS7RjzZC0QbpN8zhCyXi760DyNXzgqrB9ydJr5w@mail.gmail.com","threadId":"36669","inReplyTo":"CAL=YDWkdjjMmVNcXHYr2CGsR4-eU0aHHR5Ek6JipL-Xhi=p7-w@mail.gmail.com","subject":"Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-16T15:15:47Z","receivedAt":"2014-05-16T15:15:47Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Fri, May 16, 2014 at 8:02 AM, Ronnie Sahlberg wrote:\n> On Thu, May 15, 2014 at 5:20 PM, Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n>>\n>>> Change ref_transaction_commit so that it does not free the transaction.\n>>> Instead require that a caller will end a transaction by either calling\n>>> ref_transaction_rollback or ref_transaction_free.\n>>\n>> Can I always use ref_transaction_rollback instead of ref_transaction_free?\n>> It would be convenient if my cleanup code could always call _rollback\n>> instead of having to do something different for success and errors.\n>\n> Currently, yes.\n>\n> But it might make sense to change these so rollback only clears the\n> updates that are in flight from the transaction and\n> free only frees the transaction itself iff there are no updates in flight.\n>\n> I.e. the success and error would then differ like this :\n> ...\n> if (transaction_commit()) {\n> transaction_rollback()\n> transaction_free()\n> return error(\"some error\")\n> }\n> transaction_free()\n>\n\nBut we do not really need rollback right now. If / when we decide to\nneed re-startable/re-usable transactions we can add it back when\nneeded.\n\nLet me update the patch series and remove transaction_rollback and\nreplace all calls to it with calls to transaction_free instead.\n\n\n>\n>\n>>\n>> Another way to ask the question: what is it valid to do with a\n>> transaction after commiting?\n>\n> Right now the only valid thing to do is either rollback or free. But\n> we could allow other things too :\n>\n>\n> re-usable transactions.\n> ---------------------------------\n> I don't know if this is a good reason or not, but one reason we might\n> want to keep\n> two different names could be if we want to start allowing to re-use\n> transactions.\n> For example for cases/backends where transaction_begin() might be very\n> expensive.\n>\n> For that case I would imagine we could allow to do things such as\n>\n> t = transaction_begin()\n> ...\n> /* first transaction */\n> transaction_update(...)\n> transaction_commit(...)\n> if transaction failed transaction_rollback(...)\n>\n> /* second transaction, first transaction cleared all updates in\n> flight either through commit or through rollback */\n> transaction_update()\n> transaction_commit()\n> if transaction failed transaction_rollback(...)\n> ...\n> transaction_free()\n>\n>\n> (In order to do something like this we would still need to do some\n> changes so that rollback will only free the updates that were in\n> flight but not free the transaction itself.)\n>\n>\n>>\n>> Thanks,\n>> Jonathan\n"},{"id":"241988","messageId":"20140516183348.GC12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-23-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-16T18:33:48Z","receivedAt":"2014-05-16T18:33:48Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case\n from long ago)\nRonnie Sahlberg wrote:\n\n> In s_update_ref there are two calls that when they fail we return an error\n> based on the errno value. In particular we want to return a specific error\n> if ENOTDIR happened. Both these functions do have failure modes where they\n> may return an error without updating errno\n\nThat's a bug. Any function for which errno is supposed to be\nmeaningful when it returns an error should always set errno.\nOtherwise errno may be set to ENOTDIR within the function by an\nunrelated system call.\n\nFunctions that fail and lead callers to check errno, based on a quick\nsearch with 'git grep -e 'errno *[!=]=':\n\n fopen\n lstat\n builtin/apply.c::try_create_file (= mkdir, symlink, or open)\n rmdir\n open\n mkdir\n unlink\n lock_any_ref_for_update \n write_ref_sha1\n strtol\n kill\n odb_pack_keep\n opendir\n fgets\n read\n poll\n pread\n strtoimax\n strtoumax\n git_parse_int\n git_parse_int64\n git_parse_ulong\n write_in_full\n credential-cache.c::send_request\n fstat\n run_command_v_opt\n git.c::run_argv\n readlink\n resolve_ref_unsafe\n hold_lock_file_for_update\n unlink_or_warn\n rename\n execvp\n waitpid\n execv_git_cmd\n execv_shell_cmd\n sane_execvp\n stat\n read_object\n git_mkstemp_mode\n create_tmpfile\n start_command\n xwrite\n iconv\n fast_export_ls\n fast_export_ls_rev\n delete_ref\n\nlock_any_ref_for_update has failure paths\n * check_ref_format [!]\n * lock_ref_sha1_basic\n\nlock_ref_sha1_basic has failure paths\n * remove_empty_directories\n * resolve_ref_unsafe\n * safe_create_leading_directories\n * verify_lock\n\nremove_empty_directories has one failure path\n * remove_dir_recursively\nbut could be more explicit about the need to preserve errno.\n\nerrno from remove_dir_recursively is meaningful.\n\nresolve_ref_unsafe has failure paths\n * check_refname_format [!]\n * too much recursion [!]\n * lstat, readlink, open\n * handle_missing_loose_ref\n * read_in_full, but errno gets clobbered\n * invalid ref [!]\n * invalid symref [!]\n\nverify_lock has failure paths\n * read_ref_full, but errno gets clobbered\n * old_sha1 didn't match [!]\n \nwrite_ref_sha1 has failure paths\n * !lock [!]\n * missing object [!]\n * non-commit object [!]\n * write_in_full, close_ref, but errno gets clobbered\n * log_ref_write\n * commit_ref\n\nlog_ref_write has failure paths\n * log_ref_setup\n * write_in_full, close. errno gets clobbered\n\ncommit_ref just calls commit_lock_file.\n\nlog_ref_setup has failure paths\n * safe_create_leading_directories, but errno gets clobbered\n * remove_empty_directories, but errno gets clobbered\n * open, but errno gets clobbered\n\nsafe_create_leading_directories doesn't set errno for SCLD_EXISTS\nbut otherwise its errno is useful.\n\nThat should cover the cases affecting the code path in fetch.c you\nmentioned (I haven't checked the others).\n\nHow about something like this?\n\nIt's also tempting to teach vreportf and vwritef to save errno, which\nwould handle some of these cases automatically.\n\ndiff --git i/refs.c w/refs.c\nindex 82a8d4e..f98acf0 100644\n--- i/refs.c\n+++ w/refs.c\n@@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n \tif (flag)\n \t\t*flag = 0;\n \n-\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\terrno = EINVAL;\n \t\treturn NULL;\n+\t}\n \n \tfor (;;) {\n \t\tchar path[PATH_MAX];\n@@ -1342,8 +1344,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n \t\tchar *buf;\n \t\tint fd;\n \n-\t\tif (--depth < 0)\n+\t\tif (--depth < 0) {\n+\t\t\terrno = ELOOP;\n \t\t\treturn NULL;\n+\t\t}\n \n \t\tgit_snpath(path, sizeof(path), \"%s\", refname);\n \n@@ -1405,9 +1409,13 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n \t\t\t\treturn NULL;\n \t\t}\n \t\tlen = read_in_full(fd, buffer, sizeof(buffer)-1);\n-\t\tclose(fd);\n-\t\tif (len < 0)\n+\t\tif (len < 0) {\n+\t\t\tint save_errno = errno;\n+\t\t\tclose(fd);\n+\t\t\terrno = save_errno;\n \t\t\treturn NULL;\n+\t\t}\n+\t\tclose(fd);\n \t\twhile (len && isspace(buffer[len-1]))\n \t\t\tlen--;\n \t\tbuffer[len] = '\\0';\n@@ -1424,6 +1432,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n \t\t\t (buffer[40] != '\\0' && !isspace(buffer[40]))) {\n \t\t\t\tif (flag)\n \t\t\t\t\t*flag |= REF_ISBROKEN;\n+\t\t\t\terrno = EINVAL;\n \t\t\t\treturn NULL;\n \t\t\t}\n \t\t\treturn refname;\n@@ -1436,6 +1445,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n \t\tif (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {\n \t\t\tif (flag)\n \t\t\t\t*flag |= REF_ISBROKEN;\n+\t\t\terrno = EINVAL;\n \t\t\treturn NULL;\n \t\t}\n \t\trefname = strcpy(refname_buffer, buf);\n@@ -1908,14 +1918,17 @@ static struct ref_lock *verify_lock(struct ref_lock *lock,\n \tconst unsigned char *old_sha1, int mustexist)\n {\n \tif (read_ref_full(lock->ref_name, lock->old_sha1, mustexist, NULL)) {\n+\t\tint save_errno = errno;\n \t\terror(\"Can't verify ref %s\", lock->ref_name);\n \t\tunlock_ref(lock);\n+\t\terrno = save_errno;\n \t\treturn NULL;\n \t}\n \tif (hashcmp(lock->old_sha1, old_sha1)) {\n \t\terror(\"Ref %s is at %s but expected %s\", lock->ref_name,\n \t\t\tsha1_to_hex(lock->old_sha1), sha1_to_hex(old_sha1));\n \t\tunlock_ref(lock);\n+\t\terrno = EBUSY;\n \t\treturn NULL;\n \t}\n \treturn lock;\n@@ -1928,15 +1941,17 @@ static int remove_empty_directories(const char *file)\n \t * only empty directories), remove them.\n \t */\n \tstruct strbuf path;\n-\tint result;\n+\tint result, save_errno;\n \n \tstrbuf_init(&path, 20);\n \tstrbuf_addstr(&path, file);\n \n \tresult = remove_dir_recursively(&path, REMOVE_DIR_EMPTY_ONLY);\n+\tsave_errno = errno;\n \n \tstrbuf_release(&path);\n \n+\terrno = save_errno;\n \treturn result;\n }\n \n@@ -2137,8 +2152,10 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n \t\t\t\t\t int flags, int *type_p)\n {\n-\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\terrno = EINVAL;\n \t\treturn NULL;\n+\t}\n \treturn lock_ref_sha1_basic(refname, old_sha1, flags, type_p);\n }\n \n@@ -2734,9 +2751,12 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)\n \t starts_with(refname, \"refs/remotes/\") ||\n \t starts_with(refname, \"refs/notes/\") ||\n \t !strcmp(refname, \"HEAD\"))) {\n-\t\tif (safe_create_leading_directories(logfile) < 0)\n-\t\t\treturn error(\"unable to create directory for %s\",\n-\t\t\t\t logfile);\n+\t\tif (safe_create_leading_directories(logfile) < 0) {\n+\t\t\tint save_errno = errno;\n+\t\t\terror(\"unable to create directory for %s\", logfile);\n+\t\t\terrno = save_errno;\n+\t\t\treturn -1;\n+\t\t}\n \t\toflags |= O_CREAT;\n \t}\n \n@@ -2747,15 +2767,21 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)\n \n \t\tif ((oflags & O_CREAT) && errno == EISDIR) {\n \t\t\tif (remove_empty_directories(logfile)) {\n-\t\t\t\treturn error(\"There are still logs under '%s'\",\n-\t\t\t\t\t logfile);\n+\t\t\t\tint save_errno = errno;\n+\t\t\t\terror(\"There are still logs under '%s'\", logfile);\n+\t\t\t\terrno = save_errno;\n+\t\t\t\treturn -1;\n \t\t\t}\n \t\t\tlogfd = open(logfile, oflags, 0666);\n \t\t}\n \n-\t\tif (logfd < 0)\n-\t\t\treturn error(\"Unable to append to %s: %s\",\n+\t\tif (logfd < 0) {\n+\t\t\tint save_errno = errno;\n+\t\t\terror(\"Unable to append to %s: %s\",\n \t\t\t\t logfile, strerror(errno));\n+\t\t\terrno = save_errno;\n+\t\t\treturn -1;\n+\t\t}\n \t}\n \n \tadjust_shared_perm(logfile);\n@@ -2795,8 +2821,19 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,\n \t\tlen += copy_msg(logrec + len - 1, msg) - 1;\n \twritten = len <= maxlen ? write_in_full(logfd, logrec, len) : -1;\n \tfree(logrec);\n-\tif (close(logfd) != 0 || written != len)\n-\t\treturn error(\"Unable to append to %s\", log_file);\n+\tif (written != len) {\n+\t\tint save_errno = errno;\n+\t\tclose(logfd);\n+\t\terror(\"Unable to append to %s\", log_file);\n+\t\terrno = save_errno;\n+\t\treturn -1;\n+\t}\n+\tif (close(logfd)) {\n+\t\tint save_errno = errno;\n+\t\terror(\"Unable to append to %s\", log_file);\n+\t\terrno = save_errno;\n+\t\treturn -1;\n+\t}\n \treturn 0;\n }\n \n@@ -2811,8 +2848,10 @@ int write_ref_sha1(struct ref_lock *lock,\n \tstatic char term = '\\n';\n \tstruct object *o;\n \n-\tif (!lock)\n+\tif (!lock) {\n+\t\terrno = EINVAL;\n \t\treturn -1;\n+\t}\n \tif (!lock->force_write && !hashcmp(lock->old_sha1, sha1)) {\n \t\tunlock_ref(lock);\n \t\treturn 0;\n@@ -2822,26 +2861,32 @@ int write_ref_sha1(struct ref_lock *lock,\n \t\terror(\"Trying to write ref %s with nonexistent object %s\",\n \t\t\tlock->ref_name, sha1_to_hex(sha1));\n \t\tunlock_ref(lock);\n+\t\terrno = EINVAL;\n \t\treturn -1;\n \t}\n \tif (o->type != OBJ_COMMIT && is_branch(lock->ref_name)) {\n \t\terror(\"Trying to write non-commit object %s to branch %s\",\n \t\t\tsha1_to_hex(sha1), lock->ref_name);\n \t\tunlock_ref(lock);\n+\t\terrno = EINVAL;\n \t\treturn -1;\n \t}\n \tif (write_in_full(lock->lock_fd, sha1_to_hex(sha1), 40) != 40 ||\n-\t write_in_full(lock->lock_fd, &term, 1) != 1\n-\t\t|| close_ref(lock) < 0) {\n+\t write_in_full(lock->lock_fd, &term, 1) != 1 ||\n+\t close_ref(lock) < 0) {\n+\t\tint save_errno = errno;\n \t\terror(\"Couldn't write %s\", lock->lk->filename);\n \t\tunlock_ref(lock);\n+\t\terrno = save_errno;\n \t\treturn -1;\n \t}\n \tclear_loose_ref_cache(&ref_cache);\n \tif (log_ref_write(lock->ref_name, lock->old_sha1, sha1, logmsg) < 0 ||\n \t (strcmp(lock->ref_name, lock->orig_ref_name) &&\n \t log_ref_write(lock->orig_ref_name, lock->old_sha1, sha1, logmsg) < 0)) {\n+\t\tint save_errno = errno;\n \t\tunlock_ref(lock);\n+\t\terrno = save_errno;\n \t\treturn -1;\n \t}\n \tif (strcmp(lock->orig_ref_name, \"HEAD\") != 0) {\n@@ -2866,8 +2911,10 @@ int write_ref_sha1(struct ref_lock *lock,\n \t\t\tlog_ref_write(\"HEAD\", lock->old_sha1, sha1, logmsg);\n \t}\n \tif (commit_ref(lock)) {\n+\t\tint save_errno = errno;\n \t\terror(\"Couldn't set %s\", lock->ref_name);\n \t\tunlock_ref(lock);\n+\t\terrno = save_errno;\n \t\treturn -1;\n \t}\n \tunlock_ref(lock);\ndiff --git i/sha1_file.c w/sha1_file.c\nindex 3e9f55f..a7bbba7 100644\n--- i/sha1_file.c\n+++ w/sha1_file.c\n@@ -136,8 +136,10 @@ enum scld_error safe_create_leading_directories(char *path)\n \t\t*slash = '\\0';\n \t\tif (!stat(path, &st)) {\n \t\t\t/* path exists */\n-\t\t\tif (!S_ISDIR(st.st_mode))\n+\t\t\tif (!S_ISDIR(st.st_mode)) {\n+\t\t\t\terrno = EEXIST;\n \t\t\t\tret = SCLD_EXISTS;\n+\t\t\t}\n \t\t} else if (mkdir(path, 0777)) {\n \t\t\tif (errno == EEXIST &&\n \t\t\t !stat(path, &st) && S_ISDIR(st.st_mode))\n"},{"id":"241996","messageId":"20140516191231.GE12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-24-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-16T19:12:31Z","receivedAt":"2014-05-16T19:12:31Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: peff for STORE_REF_ERROR_DF_CONFLICT expertise)\nRonnie Sahlberg wrote:\n\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -375,7 +375,7 @@ static int s_update_ref(const char *action,\n[...]\n> +\ttransaction = ref_transaction_begin();\n> +\tif (!transaction ||\n> +\t ref_transaction_update(transaction, ref->name, ref->new_sha1,\n> +\t\t\t\t ref->old_sha1, 0, check_old) ||\n> +\t ref_transaction_commit(transaction, msg, NULL)) {\n> +\t\tref_transaction_rollback(transaction);\n> \t\treturn errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n> \t\t\t\t\t STORE_REF_ERROR_OTHER;\n> +\t}\n\nI'd rather not rely on errno here (see the previous patch for why).\nIs there some other way to distinguish the case where a ref couldn't\nbe created because there was a prefix of that ref in the way?\n\nFor example, maybe ref_transaction_commit could return a different\nnegative integer in this case.\n\nThanks,\nJonathan\n"},{"id":"241997","messageId":"CAL=YDWknwYDLCRUzev6BVvfaYnnHLKoVsNaC2uJd-dTBy5QMEA@mail.gmail.com","threadId":"36669","inReplyTo":"20140516183348.GC12314@google.com","subject":"Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-16T20:26:23Z","receivedAt":"2014-05-16T20:26:23Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"errno is hairy :-(\n\n\nYou probably also want something like this :\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4603cb6..55e7dd8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -426,7 +426,7 @@ static int update_local_ref(struct ref *ref,\n _(\"! %-*s %-*s -> %s (can't fetch in current branch\n TRANSPORT_SUMMARY(_(\"[rejected]\")),\n REFCOL_WIDTH, remote, pretty_ref);\n- return 1;\n+ return STORE_REF_ERROR_OTHER;\n }\n\n if (!is_null_sha1(ref->old_sha1) &&\n@@ -513,7 +513,7 @@ static int update_local_ref(struct ref *ref,\n TRANSPORT_SUMMARY(_(\"[rejected]\")),\n REFCOL_WIDTH, remote, pretty_ref,\n _(\"(non-fast-forward)\"));\n- return 1;\n+ return STORE_REF_ERROR_OTHER;\n }\n }\n\n\nTo make it more clear that this function returns a specific error and\nnot just a generic not-zero.\n\n\nOn Fri, May 16, 2014 at 11:33 AM, Jonathan Nieder wrote:\n> (cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case\n> from long ago)\n> Ronnie Sahlberg wrote:\n>\n>> In s_update_ref there are two calls that when they fail we return an error\n>> based on the errno value. In particular we want to return a specific error\n>> if ENOTDIR happened. Both these functions do have failure modes where they\n>> may return an error without updating errno\n>\n> That's a bug. Any function for which errno is supposed to be\n> meaningful when it returns an error should always set errno.\n> Otherwise errno may be set to ENOTDIR within the function by an\n> unrelated system call.\n>\n> Functions that fail and lead callers to check errno, based on a quick\n> search with 'git grep -e 'errno *[!=]=':\n>\n> fopen\n> lstat\n> builtin/apply.c::try_create_file (= mkdir, symlink, or open)\n> rmdir\n> open\n> mkdir\n> unlink\n> lock_any_ref_for_update\n> write_ref_sha1\n> strtol\n> kill\n> odb_pack_keep\n> opendir\n> fgets\n> read\n> poll\n> pread\n> strtoimax\n> strtoumax\n> git_parse_int\n> git_parse_int64\n> git_parse_ulong\n> write_in_full\n> credential-cache.c::send_request\n> fstat\n> run_command_v_opt\n> git.c::run_argv\n> readlink\n> resolve_ref_unsafe\n> hold_lock_file_for_update\n> unlink_or_warn\n> rename\n> execvp\n> waitpid\n> execv_git_cmd\n> execv_shell_cmd\n> sane_execvp\n> stat\n> read_object\n> git_mkstemp_mode\n> create_tmpfile\n> start_command\n> xwrite\n> iconv\n> fast_export_ls\n> fast_export_ls_rev\n> delete_ref\n>\n> lock_any_ref_for_update has failure paths\n> * check_ref_format [!]\n> * lock_ref_sha1_basic\n>\n> lock_ref_sha1_basic has failure paths\n> * remove_empty_directories\n> * resolve_ref_unsafe\n> * safe_create_leading_directories\n> * verify_lock\n>\n> remove_empty_directories has one failure path\n> * remove_dir_recursively\n> but could be more explicit about the need to preserve errno.\n>\n> errno from remove_dir_recursively is meaningful.\n>\n> resolve_ref_unsafe has failure paths\n> * check_refname_format [!]\n> * too much recursion [!]\n> * lstat, readlink, open\n> * handle_missing_loose_ref\n> * read_in_full, but errno gets clobbered\n> * invalid ref [!]\n> * invalid symref [!]\n>\n> verify_lock has failure paths\n> * read_ref_full, but errno gets clobbered\n> * old_sha1 didn't match [!]\n>\n> write_ref_sha1 has failure paths\n> * !lock [!]\n> * missing object [!]\n> * non-commit object [!]\n> * write_in_full, close_ref, but errno gets clobbered\n> * log_ref_write\n> * commit_ref\n>\n> log_ref_write has failure paths\n> * log_ref_setup\n> * write_in_full, close. errno gets clobbered\n>\n> commit_ref just calls commit_lock_file.\n>\n> log_ref_setup has failure paths\n> * safe_create_leading_directories, but errno gets clobbered\n> * remove_empty_directories, but errno gets clobbered\n> * open, but errno gets clobbered\n>\n> safe_create_leading_directories doesn't set errno for SCLD_EXISTS\n> but otherwise its errno is useful.\n>\n> That should cover the cases affecting the code path in fetch.c you\n> mentioned (I haven't checked the others).\n>\n> How about something like this?\n>\n> It's also tempting to teach vreportf and vwritef to save errno, which\n> would handle some of these cases automatically.\n>\n> diff --git i/refs.c w/refs.c\n> index 82a8d4e..f98acf0 100644\n> --- i/refs.c\n> +++ w/refs.c\n> @@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> if (flag)\n> *flag = 0;\n>\n> - if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n> + if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n> + errno = EINVAL;\n> return NULL;\n> + }\n>\n> for (;;) {\n> char path[PATH_MAX];\n> @@ -1342,8 +1344,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> char *buf;\n> int fd;\n>\n> - if (--depth < 0)\n> + if (--depth < 0) {\n> + errno = ELOOP;\n> return NULL;\n> + }\n>\n> git_snpath(path, sizeof(path), \"%s\", refname);\n>\n> @@ -1405,9 +1409,13 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> return NULL;\n> }\n> len = read_in_full(fd, buffer, sizeof(buffer)-1);\n> - close(fd);\n> - if (len < 0)\n> + if (len < 0) {\n> + int save_errno = errno;\n> + close(fd);\n> + errno = save_errno;\n> return NULL;\n> + }\n> + close(fd);\n> while (len && isspace(buffer[len-1]))\n> len--;\n> buffer[len] = '\\0';\n> @@ -1424,6 +1432,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> (buffer[40] != '\\0' && !isspace(buffer[40]))) {\n> if (flag)\n> *flag |= REF_ISBROKEN;\n> + errno = EINVAL;\n> return NULL;\n> }\n> return refname;\n> @@ -1436,6 +1445,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> if (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {\n> if (flag)\n> *flag |= REF_ISBROKEN;\n> + errno = EINVAL;\n> return NULL;\n> }\n> refname = strcpy(refname_buffer, buf);\n> @@ -1908,14 +1918,17 @@ static struct ref_lock *verify_lock(struct ref_lock *lock,\n> const unsigned char *old_sha1, int mustexist)\n> {\n> if (read_ref_full(lock->ref_name, lock->old_sha1, mustexist, NULL)) {\n> + int save_errno = errno;\n> error(\"Can't verify ref %s\", lock->ref_name);\n> unlock_ref(lock);\n> + errno = save_errno;\n> return NULL;\n> }\n> if (hashcmp(lock->old_sha1, old_sha1)) {\n> error(\"Ref %s is at %s but expected %s\", lock->ref_name,\n> sha1_to_hex(lock->old_sha1), sha1_to_hex(old_sha1));\n> unlock_ref(lock);\n> + errno = EBUSY;\n> return NULL;\n> }\n> return lock;\n> @@ -1928,15 +1941,17 @@ static int remove_empty_directories(const char *file)\n> * only empty directories), remove them.\n> */\n> struct strbuf path;\n> - int result;\n> + int result, save_errno;\n>\n> strbuf_init(&path, 20);\n> strbuf_addstr(&path, file);\n>\n> result = remove_dir_recursively(&path, REMOVE_DIR_EMPTY_ONLY);\n> + save_errno = errno;\n>\n> strbuf_release(&path);\n>\n> + errno = save_errno;\n> return result;\n> }\n>\n> @@ -2137,8 +2152,10 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,\n> const unsigned char *old_sha1,\n> int flags, int *type_p)\n> {\n> - if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n> + if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n> + errno = EINVAL;\n> return NULL;\n> + }\n> return lock_ref_sha1_basic(refname, old_sha1, flags, type_p);\n> }\n>\n> @@ -2734,9 +2751,12 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)\n> starts_with(refname, \"refs/remotes/\") ||\n> starts_with(refname, \"refs/notes/\") ||\n> !strcmp(refname, \"HEAD\"))) {\n> - if (safe_create_leading_directories(logfile) < 0)\n> - return error(\"unable to create directory for %s\",\n> - logfile);\n> + if (safe_create_leading_directories(logfile) < 0) {\n> + int save_errno = errno;\n> + error(\"unable to create directory for %s\", logfile);\n> + errno = save_errno;\n> + return -1;\n> + }\n> oflags |= O_CREAT;\n> }\n>\n> @@ -2747,15 +2767,21 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)\n>\n> if ((oflags & O_CREAT) && errno == EISDIR) {\n> if (remove_empty_directories(logfile)) {\n> - return error(\"There are still logs under '%s'\",\n> - logfile);\n> + int save_errno = errno;\n> + error(\"There are still logs under '%s'\", logfile);\n> + errno = save_errno;\n> + return -1;\n> }\n> logfd = open(logfile, oflags, 0666);\n> }\n>\n> - if (logfd < 0)\n> - return error(\"Unable to append to %s: %s\",\n> + if (logfd < 0) {\n> + int save_errno = errno;\n> + error(\"Unable to append to %s: %s\",\n> logfile, strerror(errno));\n> + errno = save_errno;\n> + return -1;\n> + }\n> }\n>\n> adjust_shared_perm(logfile);\n> @@ -2795,8 +2821,19 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,\n> len += copy_msg(logrec + len - 1, msg) - 1;\n> written = len <= maxlen ? write_in_full(logfd, logrec, len) : -1;\n> free(logrec);\n> - if (close(logfd) != 0 || written != len)\n> - return error(\"Unable to append to %s\", log_file);\n> + if (written != len) {\n> + int save_errno = errno;\n> + close(logfd);\n> + error(\"Unable to append to %s\", log_file);\n> + errno = save_errno;\n> + return -1;\n> + }\n> + if (close(logfd)) {\n> + int save_errno = errno;\n> + error(\"Unable to append to %s\", log_file);\n> + errno = save_errno;\n> + return -1;\n> + }\n> return 0;\n> }\n>\n> @@ -2811,8 +2848,10 @@ int write_ref_sha1(struct ref_lock *lock,\n> static char term = '\\n';\n> struct object *o;\n>\n> - if (!lock)\n> + if (!lock) {\n> + errno = EINVAL;\n> return -1;\n> + }\n> if (!lock->force_write && !hashcmp(lock->old_sha1, sha1)) {\n> unlock_ref(lock);\n> return 0;\n> @@ -2822,26 +2861,32 @@ int write_ref_sha1(struct ref_lock *lock,\n> error(\"Trying to write ref %s with nonexistent object %s\",\n> lock->ref_name, sha1_to_hex(sha1));\n> unlock_ref(lock);\n> + errno = EINVAL;\n> return -1;\n> }\n> if (o->type != OBJ_COMMIT && is_branch(lock->ref_name)) {\n> error(\"Trying to write non-commit object %s to branch %s\",\n> sha1_to_hex(sha1), lock->ref_name);\n> unlock_ref(lock);\n> + errno = EINVAL;\n> return -1;\n> }\n> if (write_in_full(lock->lock_fd, sha1_to_hex(sha1), 40) != 40 ||\n> - write_in_full(lock->lock_fd, &term, 1) != 1\n> - || close_ref(lock) < 0) {\n> + write_in_full(lock->lock_fd, &term, 1) != 1 ||\n> + close_ref(lock) < 0) {\n> + int save_errno = errno;\n> error(\"Couldn't write %s\", lock->lk->filename);\n> unlock_ref(lock);\n> + errno = save_errno;\n> return -1;\n> }\n> clear_loose_ref_cache(&ref_cache);\n> if (log_ref_write(lock->ref_name, lock->old_sha1, sha1, logmsg) < 0 ||\n> (strcmp(lock->ref_name, lock->orig_ref_name) &&\n> log_ref_write(lock->orig_ref_name, lock->old_sha1, sha1, logmsg) < 0)) {\n> + int save_errno = errno;\n> unlock_ref(lock);\n> + errno = save_errno;\n> return -1;\n> }\n> if (strcmp(lock->orig_ref_name, \"HEAD\") != 0) {\n> @@ -2866,8 +2911,10 @@ int write_ref_sha1(struct ref_lock *lock,\n> log_ref_write(\"HEAD\", lock->old_sha1, sha1, logmsg);\n> }\n> if (commit_ref(lock)) {\n> + int save_errno = errno;\n> error(\"Couldn't set %s\", lock->ref_name);\n> unlock_ref(lock);\n> + errno = save_errno;\n> return -1;\n> }\n> unlock_ref(lock);\n> diff --git i/sha1_file.c w/sha1_file.c\n> index 3e9f55f..a7bbba7 100644\n> --- i/sha1_file.c\n> +++ w/sha1_file.c\n> @@ -136,8 +136,10 @@ enum scld_error safe_create_leading_directories(char *path)\n> *slash = '\\0';\n> if (!stat(path, &st)) {\n> /* path exists */\n> - if (!S_ISDIR(st.st_mode))\n> + if (!S_ISDIR(st.st_mode)) {\n> + errno = EEXIST;\n> ret = SCLD_EXISTS;\n> + }\n> } else if (mkdir(path, 0777)) {\n> if (errno == EEXIST &&\n> !stat(path, &st) && S_ISDIR(st.st_mode))\n"},{"id":"242004","messageId":"CAL=YDW=0DA_pxrD75n-MN1N5QyEJV36w5HkvoBUC04gZyaxNUw@mail.gmail.com","threadId":"36669","inReplyTo":"20140516191231.GE12314@google.com","subject":"Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-16T22:22:15Z","receivedAt":"2014-05-16T22:22:15Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Fri, May 16, 2014 at 12:12 PM, Jonathan Nieder wrote:\n> (+cc: peff for STORE_REF_ERROR_DF_CONFLICT expertise)\n> Ronnie Sahlberg wrote:\n>\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -375,7 +375,7 @@ static int s_update_ref(const char *action,\n> [...]\n>> + transaction = ref_transaction_begin();\n>> + if (!transaction ||\n>> + ref_transaction_update(transaction, ref->name, ref->new_sha1,\n>> + ref->old_sha1, 0, check_old) ||\n>> + ref_transaction_commit(transaction, msg, NULL)) {\n>> + ref_transaction_rollback(transaction);\n>> return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n>> STORE_REF_ERROR_OTHER;\n>> + }\n>\n> I'd rather not rely on errno here (see the previous patch for why).\n> Is there some other way to distinguish the case where a ref couldn't\n> be created because there was a prefix of that ref in the way?\n>\n> For example, maybe ref_transaction_commit could return a different\n> negative integer in this case.\n\nI have changed it to make transaction_commit will return a special\nerror if there is a name conflict\nand fetch.c now uses that instead of looking at errno to decide if to\nprint the \"try pruning\" message.\n\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"242010","messageId":"20140516225211.GI12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-25-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-16T22:52:11Z","receivedAt":"2014-05-16T22:52:11Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change store_updated_refs to use a single ref transaction for all refs that\n> are updated during the fetch. This makes the fetch more atomic when update\n> failures occur.\n\nFun.\n\n[...]\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n[...]\n> @@ -373,27 +374,13 @@ static int s_update_ref(const char *action,\n> \t\t\tstruct ref *ref,\n> \t\t\tint check_old)\n> {\n> -\tchar msg[1024];\n> -\tchar *rla = getenv(\"GIT_REFLOG_ACTION\");\n> -\tstruct ref_transaction *transaction;\n\nPreviously fetch respected GIT_REFLOG_ACTION, and this is dropping\nthat support. Intended?\n\n[...]\n> +\tif (ref_transaction_update(transaction, ref->name, ref->new_sha1,\n> +\t\t\t ref->old_sha1, 0, check_old))\n> +\t\treturn STORE_REF_ERROR_OTHER;\n\nShould pass a strbuf in to get a message back.\n\n[...]\n> +\n> \treturn 0;\n> }\n> \n> @@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \t\tgoto abort;\n> \t}\n> \n> +\terrno = 0;\n> +\ttransaction = ref_transaction_begin();\n> +\tif (!transaction) {\n> +\t\trc = error(_(\"cannot start ref transaction\\n\"));\n> +\t\tgoto abort;\n\nerror() appends a newline on its own, so the message shouldn't\nend with newline.\n\nMore importantly, this message isn't helpful without more detail about\nwhy _transaction_begin() failed. The user doesn't know what\n\n\terror: cannot start ref transaction\n\nmeans.\n\n\terror: cannot connect to mysqld for ref storage: [etc]\n\nwould tell what they need to know. That is:\n\n\ttransaction = ref_transaction_begin(err);\n\tif (!transaction) {\n\t\trc = error(\"%s\", err.buf);\n\t\tgoto abort;\n\t}\n\nerrno is not used here, so no need to set errno = 0.\n\n[...]\n> @@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \t\t\t}\n> \t\t}\n> \t}\n> +\tif ((ret = ref_transaction_commit(transaction, NULL)))\n> +\t\trc |= ret == -2 ? STORE_REF_ERROR_DF_CONFLICT :\n> +\t\t\t\tSTORE_REF_ERROR_OTHER;\n\n(I cheated and stole the newer code from your repo.)\n\nYay! Style nit: git avoids assignments in \"if\" conditionals.\n\n\tret = ref_tr...\n\tif (ret == -2)\n\t\trc |= ...\n\telse if (ret)\n\t\trc |= ...\n\nDoes _update need the too as futureproofing for backends that can\ndetect the name conflict sooner?\n\nThanks,\nJonathan\n"},{"id":"242009","messageId":"20140516225421.GJ12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-24-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-16T22:54:21Z","receivedAt":"2014-05-16T22:54:21Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n[...]\n> @@ -384,15 +384,16 @@ static int s_update_ref(const char *action,\n> \tsnprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n> \n> \terrno = 0;\n> -\tlock = lock_any_ref_for_update(ref->name,\n> -\t\t\t\t check_old ? ref->old_sha1 : NULL,\n> -\t\t\t\t 0, NULL);\n> -\tif (!lock)\n> -\t\treturn errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n> -\t\t\t\t\t STORE_REF_ERROR_OTHER;\n> -\tif (write_ref_sha1(lock, ref->new_sha1, msg) < 0)\n> +\ttransaction = ref_transaction_begin();\n> +\tif (!transaction ||\n> +\t ref_transaction_update(transaction, ref->name, ref->new_sha1,\n> +\t\t\t\t ref->old_sha1, 0, check_old) ||\n> +\t ref_transaction_commit(transaction, msg, NULL)) {\n\nSince 'err' is NULL, does that mean there's no message shown to the\nuser on error?\n"},{"id":"242012","messageId":"20140516230450.GB3988@sigill.intra.peff.net","threadId":"36669","inReplyTo":"20140516183348.GC12314@google.com","subject":"Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-05-16T23:04:50Z","receivedAt":"2014-05-16T23:04:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 16, 2014 at 11:33:48AM -0700, Jonathan Nieder wrote:\n\n> (cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case\n> from long ago)\n\nNo, but I have very good tools for searching the list archive. ;)\n\n> > In s_update_ref there are two calls that when they fail we return an error\n> > based on the errno value. In particular we want to return a specific error\n> > if ENOTDIR happened. Both these functions do have failure modes where they\n> > may return an error without updating errno\n> \n> That's a bug. Any function for which errno is supposed to be\n> meaningful when it returns an error should always set errno.\n> Otherwise errno may be set to ENOTDIR within the function by an\n> unrelated system call.\n\nI'd agree. It's OK for a function not to set errno if it is\n_successful_. But if setting errno on error is part of the interface for\nlock_any_ref_for_update, it should do so consistently.\n\nWe have not been very good about documenting which functions use errno,\nor using it consistently (e.g., in my original patch, we are explicitly\nconverting errno into a numeric return value!), so I can definitely see\nhow it is confusing.\n\n> That should cover the cases affecting the code path in fetch.c you\n> mentioned (I haven't checked the others).\n> \n> How about something like this?\n> [...]\n\nYeah, I think this is a better direction.\n\n> diff --git i/refs.c w/refs.c\n> index 82a8d4e..f98acf0 100644\n> --- i/refs.c\n> +++ w/refs.c\n> @@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea\n> \tif (flag)\n> \t\t*flag = 0;\n> \n> -\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n> +\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n> +\t\terrno = EINVAL;\n> \t\treturn NULL;\n> +\t}\n\ncheck_refname_format comes up in a few places. I wonder if it should set\nEINVAL itself.\n\nOther than that, I just skimmed through your list. All looked\nreasonable to me, though I was not thorough enough to be sure we got\ncall cases (OTOH, we can add them later on top).\n\n-Peff\n"},{"id":"242180","messageId":"CAL=YDWkmYxTm_RJ2O0Wt0xW8MxsmyZD2nDa=4_p9wQYKFPTvVw@mail.gmail.com","threadId":"36669","inReplyTo":"20140516225211.GI12314@google.com","subject":"Re: [PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-19T16:56:43Z","receivedAt":"2014-05-19T16:56:43Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Fri, May 16, 2014 at 3:52 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Change store_updated_refs to use a single ref transaction for all refs that\n>> are updated during the fetch. This makes the fetch more atomic when update\n>> failures occur.\n>\n> Fun.\n>\n> [...]\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n> [...]\n>> @@ -373,27 +374,13 @@ static int s_update_ref(const char *action,\n>> struct ref *ref,\n>> int check_old)\n>> {\n>> - char msg[1024];\n>> - char *rla = getenv(\"GIT_REFLOG_ACTION\");\n>> - struct ref_transaction *transaction;\n>\n> Previously fetch respected GIT_REFLOG_ACTION, and this is dropping\n> that support. Intended?\n\nI think it was added back later in the series when\nref_transaction_update started taking a \"msg\" argument.\nI have reordered the patches in the ref-transactions so that the fetch\nupdates come after we add msg support to _update.\n\nPlease see current builtin/fetch.c in my ref-transactions branch.\n\n\n>\n> [...]\n>> + if (ref_transaction_update(transaction, ref->name, ref->new_sha1,\n>> + ref->old_sha1, 0, check_old))\n>> + return STORE_REF_ERROR_OTHER;\n>\n> Should pass a strbuf in to get a message back.\n\nI added err strbuf to both update and commit and now also error(\"%s\",\nerr.buf) on failure.\n\n\n>\n> [...]\n>> +\n>> return 0;\n>> }\n>>\n>> @@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n>> goto abort;\n>> }\n>>\n>> + errno = 0;\n>> + transaction = ref_transaction_begin();\n>> + if (!transaction) {\n>> + rc = error(_(\"cannot start ref transaction\\n\"));\n>> + goto abort;\n>\n> error() appends a newline on its own, so the message shouldn't\n> end with newline.\n>\n> More importantly, this message isn't helpful without more detail about\n> why _transaction_begin() failed. The user doesn't know what\n>\n> error: cannot start ref transaction\n>\n> means.\n>\n> error: cannot connect to mysqld for ref storage: [etc]\n>\n> would tell what they need to know. That is:\n>\n> transaction = ref_transaction_begin(err);\n> if (!transaction) {\n> rc = error(\"%s\", err.buf);\n> goto abort;\n> }\n>\n\nI add this in the next patch series. Right now ref_transaction_begin\ncan never return failure back to the caller.\n\n\n> errno is not used here, so no need to set errno = 0.\n\nremoved.\n\n>\n> [...]\n>> @@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n>> }\n>> }\n>> }\n>> + if ((ret = ref_transaction_commit(transaction, NULL)))\n>> + rc |= ret == -2 ? STORE_REF_ERROR_DF_CONFLICT :\n>> + STORE_REF_ERROR_OTHER;\n>\n> (I cheated and stole the newer code from your repo.)\n>\n> Yay! Style nit: git avoids assignments in \"if\" conditionals.\n>\n> ret = ref_tr...\n> if (ret == -2)\n> rc |= ...\n> else if (ret)\n> rc |= ...\n>\n> Does _update need the too as futureproofing for backends that can\n> detect the name conflict sooner?\n\nYes it will. I will update the next series to do this. If _update\nfails with name conflict it will return -2.\nAlso it will store the status in the transaction so that a later call\nto _commit will also return -2.\n\n\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"242181","messageId":"CAL=YDWm67m2KC306g2gaqEoifLFnhj+aTFQQNUEYCa+2QdBNNQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140516225421.GJ12314@google.com","subject":"Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-19T16:58:26Z","receivedAt":"2014-05-19T16:58:26Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Fri, May 16, 2014 at 3:54 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n> [...]\n>> @@ -384,15 +384,16 @@ static int s_update_ref(const char *action,\n>> snprintf(msg, sizeof(msg), \"%s: %s\", rla, action);\n>>\n>> errno = 0;\n>> - lock = lock_any_ref_for_update(ref->name,\n>> - check_old ? ref->old_sha1 : NULL,\n>> - 0, NULL);\n>> - if (!lock)\n>> - return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :\n>> - STORE_REF_ERROR_OTHER;\n>> - if (write_ref_sha1(lock, ref->new_sha1, msg) < 0)\n>> + transaction = ref_transaction_begin();\n>> + if (!transaction ||\n>> + ref_transaction_update(transaction, ref->name, ref->new_sha1,\n>> + ref->old_sha1, 0, check_old) ||\n>> + ref_transaction_commit(transaction, msg, NULL)) {\n>\n> Since 'err' is NULL, does that mean there's no message shown to the\n> user on error?\n\nYes. Updated in the ref-transactions branch.\n"},{"id":"242318","messageId":"20140520194246.GR12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-26-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-20T19:42:46Z","receivedAt":"2014-05-20T19:42:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Wrap all the ref updates inside a transaction to make the update atomic.\n\nInteresting.\n\n[...]\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -46,6 +46,8 @@ static void *head_name_to_free;\n> static int sent_capabilities;\n> static int shallow_update;\n> static const char *alt_shallow_file;\n> +static struct strbuf err = STRBUF_INIT;\n\nI think it would be cleaner for err to be local. It isn't used for\ncommunication between functions.\n\n[...]\n> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n> \t\t update_shallow_ref(cmd, si))\n> \t\t\treturn \"shallow error\";\n> \n> -\t\tlock = lock_any_ref_for_update(namespaced_name, old_sha1,\n> -\t\t\t\t\t 0, NULL);\n> -\t\tif (!lock) {\n> -\t\t\trp_error(\"failed to lock %s\", name);\n> -\t\t\treturn \"failed to lock\";\n> -\t\t}\n> -\t\tif (write_ref_sha1(lock, new_sha1, \"push\")) {\n> -\t\t\treturn \"failed to write\"; /* error() already called */\n> -\t\t}\n> +\t\tif (ref_transaction_update(transaction, namespaced_name,\n> +\t\t\t\t\t new_sha1, old_sha1, 0, 1, &err))\n> +\t\t\treturn \"failed to update\";\n\nThe original used rp_error to send an error message immediately via\nsideband. This drops that --- intended?\n\nThe old error string shown on the push status line was was \"failed to\nlock\" or \"failed to write\" which makes it clear that the cause is\ncontention or database problems or filesystem problems, respectively.\nAfter this change it would say \"failed to update\" which is about as\nclear as \"failed\".\n\nWould it be safe to send err.buf as-is over the wire, or does it\ncontain information or formatting that wouldn't be suitable for the\nclient? (I haven't thought this through completely yet.) Is there\nsome easy way to distinguish between failure to lock and failure to\nwrite? Or is there some one-size-fits-all error for this case?\n\nWhen the transaction fails, we need to make sure that all ref updates\nemit 'ng' and not 'ok' in receive-pack.c::report (see the example at\nthe end of Documentation/technical/pack-protocol.txt for what this\nmeans). What error string should they use? Is there some way to make\nit clear to the user which ref was the culprit?\n\nWhat should happen when checks outside the ref transaction system\ncause a ref update to fail? I'm thinking of\n\n * per-ref 'update' hook (see githooks(5))\n * fast-forward check\n * ref creation/deletion checks\n * attempt to push to the currently checked out branch\n\nI think the natural thing to do would be to put each ref update in its\nown transaction to start so the semantics do not change right away.\nIf there are obvious answers to all these questions, then a separate\npatch could combine all these into a single transaction; or if there\nare no obvious answers, we could make the single-transaction-per-push\nsemantics optional (using a configuration variable or protocol\ncapability or something) to make it possible to experiment.\n\nHope that helps,\nJonathan\n"},{"id":"242320","messageId":"CAL=YDWnaAJ+Ck9Mt462T2zp9KOFaDAGeXKFFH2qJmjb1gj8SbQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140520194246.GR12314@google.com","subject":"Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-20T20:37:47Z","receivedAt":"2014-05-20T20:37:47Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Tue, May 20, 2014 at 12:42 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Wrap all the ref updates inside a transaction to make the update atomic.\n>\n> Interesting.\n>\n> [...]\n>> --- a/builtin/receive-pack.c\n>> +++ b/builtin/receive-pack.c\n>> @@ -46,6 +46,8 @@ static void *head_name_to_free;\n>> static int sent_capabilities;\n>> static int shallow_update;\n>> static const char *alt_shallow_file;\n>> +static struct strbuf err = STRBUF_INIT;\n>\n> I think it would be cleaner for err to be local. It isn't used for\n> communication between functions.\n\nDone.\n\n>\n> [...]\n>> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n>> update_shallow_ref(cmd, si))\n>> return \"shallow error\";\n>>\n>> - lock = lock_any_ref_for_update(namespaced_name, old_sha1,\n>> - 0, NULL);\n>> - if (!lock) {\n>> - rp_error(\"failed to lock %s\", name);\n>> - return \"failed to lock\";\n>> - }\n>> - if (write_ref_sha1(lock, new_sha1, \"push\")) {\n>> - return \"failed to write\"; /* error() already called */\n>> - }\n>> + if (ref_transaction_update(transaction, namespaced_name,\n>> + new_sha1, old_sha1, 0, 1, &err))\n>> + return \"failed to update\";\n>\n> The original used rp_error to send an error message immediately via\n> sideband. This drops that --- intended?\n\nOops. I misread it as a normal error()\n\n>\n> The old error string shown on the push status line was was \"failed to\n> lock\" or \"failed to write\" which makes it clear that the cause is\n> contention or database problems or filesystem problems, respectively.\n> After this change it would say \"failed to update\" which is about as\n> clear as \"failed\".\n\nI changed it to return xstrdup(err.buf) which should be as detailed as\nwe can get.\n\n>\n> Would it be safe to send err.buf as-is over the wire, or does it\n> contain information or formatting that wouldn't be suitable for the\n> client? (I haven't thought this through completely yet.) Is there\n> some easy way to distinguish between failure to lock and failure to\n> write? Or is there some one-size-fits-all error for this case?\n\nI think err.buf is what we want.\n\n>\n> When the transaction fails, we need to make sure that all ref updates\n> emit 'ng' and not 'ok' in receive-pack.c::report (see the example at\n> the end of Documentation/technical/pack-protocol.txt for what this\n> means). What error string should they use? Is there some way to make\n> it clear to the user which ref was the culprit?\n>\n> What should happen when checks outside the ref transaction system\n> cause a ref update to fail? I'm thinking of\n>\n> * per-ref 'update' hook (see githooks(5))\n> * fast-forward check\n> * ref creation/deletion checks\n> * attempt to push to the currently checked out branch\n>\n> I think the natural thing to do would be to put each ref update in its\n> own transaction to start so the semantics do not change right away.\n> If there are obvious answers to all these questions, then a separate\n> patch could combine all these into a single transaction; or if there\n> are no obvious answers, we could make the single-transaction-per-push\n> semantics optional (using a configuration variable or protocol\n> capability or something) to make it possible to experiment.\n\nI changed it to use one transaction per ref for now.\n\nPlease see the update ref-transactions branch.\n\nThanks!\n"},{"id":"242321","messageId":"20140520203857.GS12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-27-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-20T20:38:57Z","receivedAt":"2014-05-20T20:38:57Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> [Subject: fast-import.c: use a ref transaction when dumping tags]\n\nThis seems like an odd thing to do: either it would make sense to have\na single transaction for all imported refs so all fail or succeed\ntogether, or there would be separate transactions for each ref.\n\nThat said, I don't mind, particularly if it's a step on the way to\nusing a single transaction for everything being dumped.\n\n[...]\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -1736,15 +1736,22 @@ static void dump_tags(void)\n> {\n> \tstatic const char *msg = \"fast-import\";\n> \tstruct tag *t;\n> -\tstruct ref_lock *lock;\n> \tchar ref_name[PATH_MAX];\n> +\tstruct strbuf err = STRBUF_INIT;\n> +\tstruct ref_transaction *transaction;\n> \n> +\ttransaction = ref_transaction_begin();\n> \tfor (t = first_tag; t; t = t->next_tag) {\n> -\t\tsprintf(ref_name, \"tags/%s\", t->name);\n> +\t\tsprintf(ref_name, \"refs/tags/%s\", t->name);\n\nCan this overflow the buffer?\n\n> -\t\tlock = lock_ref_sha1(ref_name, NULL);\n> -\t\tif (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)\n> -\t\t\tfailure |= error(\"Unable to update %s\", ref_name);\n> +\n> +\t\tif (ref_transaction_update(transaction, ref_name, t->sha1,\n> +\t\t\t\t\t NULL, 0, 0, &err)) {\n> +\t\t\tfailure |= 1;\n> +\t\t\tbreak;\n> +\t\t}\n> \t}\n> +\tif (failure || ref_transaction_commit(transaction, msg, &err))\n> +\t\tfailure |= error(\"Unable to update %s\", err.buf);\n\nThis 'if (failure || ...) failure |=' idiom seems strange.\nIs err.buf always set if failure is nonzero?\n\nElsewhere failure is 0 or -1 but this introduces the possibility for\nit to be temporarily 1.\n\ndump_branches could have caused failure to be -1 before dump_tags\nis called. I think the intent is\n\n\t\tif (ref_transaction_update(...)) {\n\t\t\tfailure |= error(...);\n\t\t\tgoto out;\n\t\t}\n\t}\n\tif (ref_transaction_commit(...))\n\t\tfailure |= error(...);\n out:\n\tref_transaction_free(transaction);\n\t...\n\nThanks,\nJonathan\n"},{"id":"242323","messageId":"CAL=YDW=6jaTom5vUTU9YK3D93dGH8rB799Yt+X6bmSAR5-ZmUw@mail.gmail.com","threadId":"36669","inReplyTo":"20140520203857.GS12314@google.com","subject":"Re: [PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-20T20:53:27Z","receivedAt":"2014-05-20T20:53:27Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Tue, May 20, 2014 at 1:38 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> [Subject: fast-import.c: use a ref transaction when dumping tags]\n>\n> This seems like an odd thing to do: either it would make sense to have\n> a single transaction for all imported refs so all fail or succeed\n> together, or there would be separate transactions for each ref.\n>\n> That said, I don't mind, particularly if it's a step on the way to\n> using a single transaction for everything being dumped.\n\nFor now they are two transactions but I will merge them into a single one later.\n\n>\n> [...]\n>> --- a/fast-import.c\n>> +++ b/fast-import.c\n>> @@ -1736,15 +1736,22 @@ static void dump_tags(void)\n>> {\n>> static const char *msg = \"fast-import\";\n>> struct tag *t;\n>> - struct ref_lock *lock;\n>> char ref_name[PATH_MAX];\n>> + struct strbuf err = STRBUF_INIT;\n>> + struct ref_transaction *transaction;\n>>\n>> + transaction = ref_transaction_begin();\n>> for (t = first_tag; t; t = t->next_tag) {\n>> - sprintf(ref_name, \"tags/%s\", t->name);\n>> + sprintf(ref_name, \"refs/tags/%s\", t->name);\n>\n> Can this overflow the buffer?\n\nChanged to snprint. (We probably should do this everywhere.)\n\n>\n>> - lock = lock_ref_sha1(ref_name, NULL);\n>> - if (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)\n>> - failure |= error(\"Unable to update %s\", ref_name);\n>> +\n>> + if (ref_transaction_update(transaction, ref_name, t->sha1,\n>> + NULL, 0, 0, &err)) {\n>> + failure |= 1;\n>> + break;\n>> + }\n>> }\n>> + if (failure || ref_transaction_commit(transaction, msg, &err))\n>> + failure |= error(\"Unable to update %s\", err.buf);\n>\n> This 'if (failure || ...) failure |=' idiom seems strange.\n> Is err.buf always set if failure is nonzero?\n>\n> Elsewhere failure is 0 or -1 but this introduces the possibility for\n> it to be temporarily 1.\n>\n> dump_branches could have caused failure to be -1 before dump_tags\n> is called. I think the intent is\n>\n> if (ref_transaction_update(...)) {\n> failure |= error(...);\n> goto out;\n> }\n> }\n> if (ref_transaction_commit(...))\n> failure |= error(...);\n> out:\n> ref_transaction_free(transaction);\n> ...\n>\n\nActually, since the recent change to make _commit fail if the\ntransaction has an error, what we want is something like this :\n\n ...\n if (ref_transaction_update(transaction, ref_name, t->sha1,\nNULL, 0, 0, &err))\n break;\n }\n if (ref_transaction_commit(transaction, msg, &err))\n failure |= error(\"%s\", err.buf);\n ...\n\n\nPlease see updated ref-transactions branch.\n\n\nThanks.\n\n> Thanks,\n> Jonathan\n"},{"id":"242348","messageId":"20140521004632.GX12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-28-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 27/44] walker.c: use ref transaction for ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T00:46:32Z","receivedAt":"2014-05-21T00:46:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This changes the locking slightly for walker_fetch. Previously the code would\n> lock all refs before writing them but now we do not lock the refs until the\n> commit stage. There is thus a very short window where changes could be done\n> locally during the fetch which would be overwritten when the fetch completes\n> and commits its transaction. But this window should be reasonably short.\n> Even if this race does trigger, since both the old code and the new code\n> just overwrites the refs to the new values without checking or comparing\n> them with the previous value, this is not too dissimilar to a similar scenario\n> where you first do a ref change locally and then later do a fetch that\n> overwrites the local change. With this in mind I do not see the change in\n> locking semantics to be critical.\n\nSounds scary. The usual approach is\n\n\told_sha1 = ...\n\t... various checks ...\n\n\ttransaction = transaction_begin(&err)\n\ttransaction_update(transaction, refname, new_sha1, old_sha1, ...);\n\ttransaction_commit(transaction, &err);\n\nwhich is not racy because _update checks against old_sha1.\n\nIf I understand correctly, you are saying 'have_old' is false here so\nwe don't have the usual protection. If the \"... various checks ...\"\nsection shown above is empty, that should be fine and there is no\nactual change in semantics. If the \"... various checks ...\" section\nshown above is nonempty then it could be a problem.\n\n[...]\n> --- a/walker.c\n> +++ b/walker.c\n> @@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)\n> int walker_fetch(struct walker *walker, int targets, char **target,\n> \t\t const char **write_ref, const char *write_ref_log_details)\n> {\n> -\tstruct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));\n> +\tchar ref_name[PATH_MAX];\n\nWe tend to prefer strbuf instead of fixed-size buffers in new code.\n\n[...]\n> -\tchar *msg;\n> +\tchar *msg = NULL;\n\nNeeded? The existing code seems to set msg = NULL in the\n!write_ref_log_details case already.\n\n[...]\n> @@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n> \tfor (i = 0; i < targets; i++) {\n> \t\tif (!write_ref || !write_ref[i])\n> \t\t\tcontinue;\n> -\t\tret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : \"fetch (unknown)\");\n\nOkay, so before this patch we do:\n\n\tfor each target in write_ref:\n\t\tlock it (with no particular expectation about where it\n\t\tpoints)\n\nThen\n\n\tunless http-fetch was passed --recover:\n\t\tmark the objects pointed to by current refs as COMPLETE\n\nThen we do HTTP GETs to grab the objects we need from a \"dumb\" HTTP\nserver. The COMPLETE objects tell us about objects we don't have to\nbother trying to get.\n\nWhen we're done, we come up with a reflog entry and write out refs\npointing to the requested commits.\n\nThis code has two callers:\n\n\tgit-remote-http (aka remote-curl.c::fetch_dumb)\n\tgit-http-fetch (aka http-fetch.c)\n\nThe codepath in git-remote-http gets wide use, though it's diminishing\nas more people switch to \"smart\" http. It doesn't 't use the \"write\nout some refs\" feature. It just wants the objects and then takes care\nof writing refs on its own.\n\nPerhaps it's worth avoiding beginning a transaction in the first place\nin the !write_ref case.\n\nThe git-http-fetch command is a piece of plumbing that used to be used\nby 'git clone' and 'git fetch' in the olden days when they were shell\nscripts. I doubt anyone uses it. As you noticed, it doesn't have any\nway to specify anything about the expected old values of the refs it\nwrites to. So this change doesn't introduce any race there.\n\n> +\t\tsprintf(ref_name, \"refs/%s\", write_ref[i]);\n> +\t\tif (ref_transaction_update(transaction, ref_name,\n> +\t\t\t\t\t &sha1[20 * i], NULL,\n> +\t\t\t\t\t 0, 0))\n> +\t\t\tgoto rollback_and_fail;\n> +\t}\n\nLooks good.\n\n> +\n> +\tif (ref_transaction_commit(transaction, msg ? msg : \"fetch (unknown)\",\n> +\t\t\t\t &err)) {\n> +\t\terror(\"%s\", err.buf);\n> +\t\tgoto rollback_and_fail;\n> \t}\n\nAlso looks good.\n\nThanks,\nJonathan\n"},{"id":"242350","messageId":"20140521005134.GY12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-29-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 28/44] refs.c: make write_ref_sha1 static","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T00:51:34Z","receivedAt":"2014-05-21T00:51:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> No external users call write_ref_sha1 any more so lets declare it static.\n\nYay!\n\n[...]\n> +++ b/refs.c\n> @@ -251,6 +251,8 @@ struct ref_entry {\n[...]\n> static void read_loose_refs(const char *dirname, struct ref_dir *dir);\n> +static int write_ref_sha1(struct ref_lock *lock,\n> +\t\t\t const unsigned char *sha1, const char *logmsg);\n\nIs this forward declaration needed?\n\n[...]\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);\n> /** Release any lock taken but not written. **/\n> extern void unlock_ref(struct ref_lock *lock);\n> \n> -/** Writes sha1 into the ref specified by the lock. **/\n> -extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);\n\n(nit) Would be nice to keep the documentation comment.\n\nThanks,\nJonathan\n"},{"id":"242351","messageId":"20140521005230.GZ12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-30-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 29/44] refs.c: make lock_ref_sha1 static","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T00:52:30Z","receivedAt":"2014-05-21T00:52:30Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> No external callers reference lock_ref_sha1 any more so lets declare it static.\n>\n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 2 +-\n> refs.h | 3 ---\n> 2 files changed, 1 insertion(+), 4 deletions(-)\n\nReviewed-by: Jonathan Nieder \n"},{"id":"242382","messageId":"CAL=YDWnyrL7L6gQKg87tj25Gh6NDp867HfJX_4j85f5wDRGh6w@mail.gmail.com","threadId":"36669","inReplyTo":"20140521005134.GY12314@google.com","subject":"Re: [PATCH v8 28/44] refs.c: make write_ref_sha1 static","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-21T14:46:39Z","receivedAt":"2014-05-21T14:46:39Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Thanks!\n\nOn Tue, May 20, 2014 at 5:51 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> No external users call write_ref_sha1 any more so lets declare it static.\n>\n> Yay!\n>\n> [...]\n>> +++ b/refs.c\n>> @@ -251,6 +251,8 @@ struct ref_entry {\n> [...]\n>> static void read_loose_refs(const char *dirname, struct ref_dir *dir);\n>> +static int write_ref_sha1(struct ref_lock *lock,\n>> + const unsigned char *sha1, const char *logmsg);\n>\n> Is this forward declaration needed?\n>\n\nNo. Removed.\n(I could have sworn I had a patch to remove this forward declaration.)\n\n> [...]\n>> --- a/refs.h\n>> +++ b/refs.h\n>> @@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);\n>> /** Release any lock taken but not written. **/\n>> extern void unlock_ref(struct ref_lock *lock);\n>>\n>> -/** Writes sha1 into the ref specified by the lock. **/\n>> -extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);\n>\n> (nit) Would be nice to keep the documentation comment.\n\nMoved this comment to refs.c\n\nPlease see ref-transactions branch\nThanks!\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"242394","messageId":"CAL=YDW=DxtPBH=zBaNNzazz=vhVnWAKpOVXucK4Ezm=O_1VOFQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140521004632.GX12314@google.com","subject":"Re: [PATCH v8 27/44] walker.c: use ref transaction for ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-21T17:06:43Z","receivedAt":"2014-05-21T17:06:43Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Tue, May 20, 2014 at 5:46 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> This changes the locking slightly for walker_fetch. Previously the code would\n>> lock all refs before writing them but now we do not lock the refs until the\n>> commit stage. There is thus a very short window where changes could be done\n>> locally during the fetch which would be overwritten when the fetch completes\n>> and commits its transaction. But this window should be reasonably short.\n>> Even if this race does trigger, since both the old code and the new code\n>> just overwrites the refs to the new values without checking or comparing\n>> them with the previous value, this is not too dissimilar to a similar scenario\n>> where you first do a ref change locally and then later do a fetch that\n>> overwrites the local change. With this in mind I do not see the change in\n>> locking semantics to be critical.\n>\n> Sounds scary. The usual approach is\n>\n> old_sha1 = ...\n> ... various checks ...\n>\n> transaction = transaction_begin(&err)\n> transaction_update(transaction, refname, new_sha1, old_sha1, ...);\n> transaction_commit(transaction, &err);\n>\n> which is not racy because _update checks against old_sha1.\n>\n> If I understand correctly, you are saying 'have_old' is false here so\n> we don't have the usual protection. If the \"... various checks ...\"\n> section shown above is empty, that should be fine and there is no\n> actual change in semantics. If the \"... various checks ...\" section\n> shown above is nonempty then it could be a problem.\n\nYeah, we don't have the old sha1 so this function is already imo a bit dodgy.\nThe transaction system in this series can not really handle this with\nthe exact same semantics as the old code\nbut we do gain the ability to do things such as\n\ntransaction_begin\n... lock all refs...\n...do other stuff.\ntransaction_update for all refs\ntransaction_commit\n\nin the next patch series when the transactions start supporting\nmultiple updates for the same ref.\nHowever that support comes reasonably late in the next series and I\nwould want to avoid growing this series much more.\n\n\nThis is the patch in the next series that restore the lock-everything-first.\nIt relies on that we will do all locking in _update already and not\nwait until _commit as well as that it allows multiple updates to the\nsame ref as long as all updates, except the first, are of the form\nhave_old==1 and old_sha1==new_sha1-of-previous-update.\n\n\ndiff --git a/walker.c b/walker.c\nindex 94d4988..dd8c11e 100644\n--- a/walker.c\n+++ b/walker.c\n@@ -255,18 +255,31 @@ int walker_fetch(struct walker *walker, int targets, char\n struct strbuf err = STRBUF_INIT;\n struct ref_transaction *transaction;\n unsigned char *sha1 = xmalloc(targets * 20);\n+ unsigned char transaction_sha1[20];\n char *msg;\n int i;\n\n save_commit_buffer = 0;\n\n if (write_ref) {\n+ memset(transaction_sha1, 0xff, 20);\n transaction = transaction_begin(&err);\n if (!transaction) {\n error(\"%s\", err.buf);\n strbuf_release(&err);\n return -1;\n }\n+ /* Lock all refs by updating to a temporary sha1 */\n+ for (i = 0; i < targets; i++) {\n+ if (!write_ref[i])\n+ continue;\n+ strbuf_reset(&ref_name);\n+ strbuf_addf(&ref_name, \"refs/%s\", write_ref[i]);\n+ transaction_update_sha1(transaction, ref_name.buf,\n+ transaction_sha1, NULL, 0, 0,\n+ msg ? msg : \"fetch (unknown)\",\n+ &err);\n+ }\n }\n if (!walker->get_recover)\n for_each_ref(mark_complete, NULL);\n@@ -295,7 +308,8 @@ int walker_fetch(struct walker *walker, int targets, char **\n strbuf_reset(&ref_name);\n strbuf_addf(&ref_name, \"refs/%s\", write_ref[i]);\n if (transaction_update_sha1(transaction, ref_name.buf,\n- &sha1[20 * i], NULL, 0, 0,\n+ &sha1[20 * i], transaction_sha1,\n+ 0, 1,\n msg ? msg : \"fetch (unknown)\",\n &err))\n break;\n\n\n\n>\n> [...]\n>> --- a/walker.c\n>> +++ b/walker.c\n>> @@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)\n>> int walker_fetch(struct walker *walker, int targets, char **target,\n>> const char **write_ref, const char *write_ref_log_details)\n>> {\n>> - struct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));\n>> + char ref_name[PATH_MAX];\n>\n> We tend to prefer strbuf instead of fixed-size buffers in new code.\n\nChanged.\n\n>\n> [...]\n>> - char *msg;\n>> + char *msg = NULL;\n>\n> Needed? The existing code seems to set msg = NULL in the\n> !write_ref_log_details case already.\n\nFixed.\n\nThanks.\n\n\n>\n> [...]\n>> @@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n>> for (i = 0; i < targets; i++) {\n>> if (!write_ref || !write_ref[i])\n>> continue;\n>> - ret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : \"fetch (unknown)\");\n>\n> Okay, so before this patch we do:\n>\n> for each target in write_ref:\n> lock it (with no particular expectation about where it\n> points)\n>\n> Then\n>\n> unless http-fetch was passed --recover:\n> mark the objects pointed to by current refs as COMPLETE\n>\n> Then we do HTTP GETs to grab the objects we need from a \"dumb\" HTTP\n> server. The COMPLETE objects tell us about objects we don't have to\n> bother trying to get.\n>\n> When we're done, we come up with a reflog entry and write out refs\n> pointing to the requested commits.\n>\n> This code has two callers:\n>\n> git-remote-http (aka remote-curl.c::fetch_dumb)\n> git-http-fetch (aka http-fetch.c)\n>\n> The codepath in git-remote-http gets wide use, though it's diminishing\n> as more people switch to \"smart\" http. It doesn't 't use the \"write\n> out some refs\" feature. It just wants the objects and then takes care\n> of writing refs on its own.\n>\n> Perhaps it's worth avoiding beginning a transaction in the first place\n> in the !write_ref case.\n>\n> The git-http-fetch command is a piece of plumbing that used to be used\n> by 'git clone' and 'git fetch' in the olden days when they were shell\n> scripts. I doubt anyone uses it. As you noticed, it doesn't have any\n> way to specify anything about the expected old values of the refs it\n> writes to. So this change doesn't introduce any race there.\n>\n>> + sprintf(ref_name, \"refs/%s\", write_ref[i]);\n>> + if (ref_transaction_update(transaction, ref_name,\n>> + &sha1[20 * i], NULL,\n>> + 0, 0))\n>> + goto rollback_and_fail;\n>> + }\n>\n> Looks good.\n>\n>> +\n>> + if (ref_transaction_commit(transaction, msg ? msg : \"fetch (unknown)\",\n>> + &err)) {\n>> + error(\"%s\", err.buf);\n>> + goto rollback_and_fail;\n>> }\n>\n> Also looks good.\n>\n> Thanks,\n> Jonathan\n"},{"id":"242403","messageId":"CAL=YDWnftEnmmKFj=6vyG9bhfEL-dO5gpRPNCD0+X38LKdi7Gg@mail.gmail.com","threadId":"36669","inReplyTo":"CAL=YDWnaAJ+Ck9Mt462T2zp9KOFaDAGeXKFFH2qJmjb1gj8SbQ@mail.gmail.com","subject":"Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-21T18:50:01Z","receivedAt":"2014-05-21T18:50:01Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Tue, May 20, 2014 at 1:37 PM, Ronnie Sahlberg wrote:\n> On Tue, May 20, 2014 at 12:42 PM, Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n>>\n>>> Wrap all the ref updates inside a transaction to make the update atomic.\n>>\n>> Interesting.\n>>\n>> [...]\n>>> --- a/builtin/receive-pack.c\n>>> +++ b/builtin/receive-pack.c\n>>> @@ -46,6 +46,8 @@ static void *head_name_to_free;\n>>> static int sent_capabilities;\n>>> static int shallow_update;\n>>> static const char *alt_shallow_file;\n>>> +static struct strbuf err = STRBUF_INIT;\n>>\n>> I think it would be cleaner for err to be local. It isn't used for\n>> communication between functions.\n>\n> Done.\n>\n>>\n>> [...]\n>>> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n>>> update_shallow_ref(cmd, si))\n>>> return \"shallow error\";\n>>>\n>>> - lock = lock_any_ref_for_update(namespaced_name, old_sha1,\n>>> - 0, NULL);\n>>> - if (!lock) {\n>>> - rp_error(\"failed to lock %s\", name);\n>>> - return \"failed to lock\";\n>>> - }\n>>> - if (write_ref_sha1(lock, new_sha1, \"push\")) {\n>>> - return \"failed to write\"; /* error() already called */\n>>> - }\n>>> + if (ref_transaction_update(transaction, namespaced_name,\n>>> + new_sha1, old_sha1, 0, 1, &err))\n>>> + return \"failed to update\";\n>>\n>> The original used rp_error to send an error message immediately via\n>> sideband. This drops that --- intended?\n>\n> Oops. I misread it as a normal error()\n>\n>>\n>> The old error string shown on the push status line was was \"failed to\n>> lock\" or \"failed to write\" which makes it clear that the cause is\n>> contention or database problems or filesystem problems, respectively.\n>> After this change it would say \"failed to update\" which is about as\n>> clear as \"failed\".\n>\n> I changed it to return xstrdup(err.buf) which should be as detailed as\n> we can get.\n>\n>>\n>> Would it be safe to send err.buf as-is over the wire, or does it\n>> contain information or formatting that wouldn't be suitable for the\n>> client? (I haven't thought this through completely yet.) Is there\n>> some easy way to distinguish between failure to lock and failure to\n>> write? Or is there some one-size-fits-all error for this case?\n>\n> I think err.buf is what we want.\n>\n>>\n>> When the transaction fails, we need to make sure that all ref updates\n>> emit 'ng' and not 'ok' in receive-pack.c::report (see the example at\n>> the end of Documentation/technical/pack-protocol.txt for what this\n>> means). What error string should they use? Is there some way to make\n>> it clear to the user which ref was the culprit?\n>>\n>> What should happen when checks outside the ref transaction system\n>> cause a ref update to fail? I'm thinking of\n>>\n>> * per-ref 'update' hook (see githooks(5))\n>> * fast-forward check\n>> * ref creation/deletion checks\n>> * attempt to push to the currently checked out branch\n>>\n>> I think the natural thing to do would be to put each ref update in its\n>> own transaction to start so the semantics do not change right away.\n>> If there are obvious answers to all these questions, then a separate\n>> patch could combine all these into a single transaction; or if there\n>> are no obvious answers, we could make the single-transaction-per-push\n>> semantics optional (using a configuration variable or protocol\n>> capability or something) to make it possible to experiment.\n>\n> I changed it to use one transaction per ref for now.\n>\n> Please see the update ref-transactions branch.\n\nI have added support for atomic pushes towards the end of the -next series.\nhttps://github.com/rsahlberg/git/tree/ref-transactions-next\n\nIt is activated by a new --atomic-push argument to send-pack and is\nthen negotiated by a new option in the wire protocol.\n\n\n>\n> Thanks!\n"},{"id":"242424","messageId":"20140521220035.GE12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-31-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T22:00:35Z","receivedAt":"2014-05-21T22:00:35Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -3308,6 +3308,12 @@ struct ref_update {\n> \tconst char refname[FLEX_ARRAY];\n> };\n> \n> +enum ref_transaction_status {\n> +\tREF_TRANSACTION_OPEN = 0,\n> +\tREF_TRANSACTION_CLOSED = 1,\n> +\tREF_TRANSACTION_ERROR = 2,\n\nWhat is the difference between _TRANSACTION_CLOSED and\n_TRANSACTION_ERROR?\n\n[...]\n> @@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)\n> \n> void ref_transaction_rollback(struct ref_transaction *transaction)\n> {\n> +\tif (!transaction)\n> +\t\treturn;\n> +\n> +\ttransaction->status = REF_TRANSACTION_ERROR;\n> +\n> \tref_transaction_free(transaction);\n\nOnce the transaction is freed, transaction->status is not reachable any\nmore so no one can tell that you've set it to _ERROR. What is the\nintended effect?\n\n[...]\n> @@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n> \tif (have_old && !old_sha1)\n> \t\tdie(\"BUG: have_old is true but old_sha1 is NULL\");\n> \n> +\tif (transaction->status != REF_TRANSACTION_OPEN)\n> +\t\tdie(\"BUG: update on transaction that is not open\");\n\nOk.\n\n[...]\n> @@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \tclear_loose_ref_cache(&ref_cache);\n> \n> cleanup:\n> +\ttransaction->status = ret ? REF_TRANSACTION_ERROR\n> +\t : REF_TRANSACTION_CLOSED;\n\nNit: odd use of whitespace.\n\nOverall thoughts: I like the idea of enforcing the API more strictly\n(\"after an error, the only permitted operations are...\"). The details\nleave me a little confused because I don't think anything is\ndistinguishing between _CLOSED and _ERROR. Maybe the enum only needs\ntwo states.\n\nThanks,\nJonathan\n"},{"id":"242425","messageId":"20140521220134.GF12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-32-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 31/44] refs.c: remove the update_ref_lock function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T22:01:34Z","receivedAt":"2014-05-21T22:01:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 30 ++++++------------------------\n> 1 file changed, 6 insertions(+), 24 deletions(-)\n\nReviewed-by: Jonathan Nieder \n"},{"id":"242427","messageId":"20140521220710.GG12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-33-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 32/44] refs.c: remove the update_ref_write function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T22:07:10Z","receivedAt":"2014-05-21T22:07:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> +++ b/refs.c\n[...]\n> @@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\tstruct ref_update *update = updates[i];\n> \n> \t\tif (!is_null_sha1(update->new_sha1)) {\n> -\t\t\tret = update_ref_write(msg,\n> -\t\t\t\t\t update->refname,\n> -\t\t\t\t\t update->new_sha1,\n> -\t\t\t\t\t update->lock, err,\n> -\t\t\t\t\t UPDATE_REFS_QUIET_ON_ERR);\n> +\t\t\tret = write_ref_sha1(update->lock, update->new_sha1,\n> +\t\t\t\t\t msg);\n\nThis changes the return value on error from 1 to -1. That seems like a\ngood change. It's probably worth mentioning in the commit message.\n"},{"id":"242428","messageId":"20140521220936.GH12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-34-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 33/44] refs.c: remove lock_ref_sha1","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T22:09:36Z","receivedAt":"2014-05-21T22:09:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 15 +++++----------\n> 1 file changed, 5 insertions(+), 10 deletions(-)\n\nHeh. :)\n\nReviewed-by: Jonathan Nieder \n"},{"id":"242429","messageId":"CAL=YDW=Vpc8XGLGUTgKQc4yP5v+ViN02EeLzgRipntskok+fbw@mail.gmail.com","threadId":"36669","inReplyTo":"20140521220035.GE12314@google.com","subject":"Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-21T22:11:29Z","receivedAt":"2014-05-21T22:11:29Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Please pull my ref-transactions branch.\n\nOn Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -3308,6 +3308,12 @@ struct ref_update {\n>> const char refname[FLEX_ARRAY];\n>> };\n>>\n>> +enum ref_transaction_status {\n>> + REF_TRANSACTION_OPEN = 0,\n>> + REF_TRANSACTION_CLOSED = 1,\n>> + REF_TRANSACTION_ERROR = 2,\n>\n> What is the difference between _TRANSACTION_CLOSED and\n> _TRANSACTION_ERROR?\n\nClosed is a transaction that has been committed successfully, and\nwhich we can not do any more updates onto.\nError is a transaction that has failed, and which we can not do any\nmore updates onto.\n\nThe distinction could be useful if in the future we add support to\nreuse a transaction\n\n>\n> [...]\n>> @@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)\n>>\n>> void ref_transaction_rollback(struct ref_transaction *transaction)\n>> {\n>> + if (!transaction)\n>> + return;\n>> +\n>> + transaction->status = REF_TRANSACTION_ERROR;\n>> +\n>> ref_transaction_free(transaction);\n>\n> Once the transaction is freed, transaction->status is not reachable any\n> more so no one can tell that you've set it to _ERROR. What is the\n> intended effect?\n\nref_transaction_rollback is no more. It has been removed.\n\n>\n> [...]\n>> @@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,\n>> if (have_old && !old_sha1)\n>> die(\"BUG: have_old is true but old_sha1 is NULL\");\n>>\n>> + if (transaction->status != REF_TRANSACTION_OPEN)\n>> + die(\"BUG: update on transaction that is not open\");\n>\n> Ok.\n>\n> [...]\n>> @@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n>> clear_loose_ref_cache(&ref_cache);\n>>\n>> cleanup:\n>> + transaction->status = ret ? REF_TRANSACTION_ERROR\n>> + : REF_TRANSACTION_CLOSED;\n>\n> Nit: odd use of whitespace.\n\nfixed in ref-transactions.\n\n\n>\n> Overall thoughts: I like the idea of enforcing the API more strictly\n> (\"after an error, the only permitted operations are...\"). The details\n> leave me a little confused because I don't think anything is\n> distinguishing between _CLOSED and _ERROR. Maybe the enum only needs\n> two states.\n\nA buggy caller might do :\n\ntransaction_begin()\ntransaction_update()\ntransaction_commit() (A)\ntransaction_update() (B)\ntransaction_commit() (C)\n\nAfter A the transaction in no longer open and until we decide we want\nto add support for re-usable transactions (which may or may not be a\ngood idea) we need to make sure that both B and C fails.\nSince the transaction in A completed successfully we can't really mark\nis as ERROR, instead we flag it as CLOSED.\n\n\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"242434","messageId":"20140521222201.GI12314@google.com","threadId":"36669","inReplyTo":"CAL=YDW=Vpc8XGLGUTgKQc4yP5v+ViN02EeLzgRipntskok+fbw@mail.gmail.com","subject":"Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T22:22:01Z","receivedAt":"2014-05-21T22:22:01Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Please pull my ref-transactions branch.\n\nI'm at bd5736cb (2014-05-21 13:46) now.\n\n> On Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n\n>>> --- a/refs.c\n>>> +++ b/refs.c\n>>> @@ -3308,6 +3308,12 @@ struct ref_update {\n>>> const char refname[FLEX_ARRAY];\n>>> };\n>>>\n>>> +enum ref_transaction_status {\n>>> + REF_TRANSACTION_OPEN = 0,\n>>> + REF_TRANSACTION_CLOSED = 1,\n>>> + REF_TRANSACTION_ERROR = 2,\n>>\n>> What is the difference between _TRANSACTION_CLOSED and\n>> _TRANSACTION_ERROR?\n>\n> Closed is a transaction that has been committed successfully, and\n> which we can not do any more updates onto.\n> Error is a transaction that has failed, and which we can not do any\n> more updates onto.\n\nThat means that both mean the caller cannot do any more updates,\nright? Why not call them both _CLOSED?\n\n> The distinction could be useful if in the future we add support to\n> reuse a transaction\n\nIf the distinction becomes useful in the future, wouldn't that\nbe the moment to add a new state?\n\nI don't mean that there has to be a big useful distinction. E.g.,\nmaybe the idea is that when using a debugger it can be useful to see\nthe difference between _ERROR and _CLOSED or something, and I think\nthat would be fine as long as the intended meaning is documented\n(e.g., using comments). My only complaint is that it's hard to\nunderstand the code and keep track of which status should be used in a\ngiven place unless there is some distinction between them.\n\n[...]\n> ref_transaction_rollback is no more. It has been removed.\n\nThanks. Sorry I missed that last time.\n\nJonathan\n"},{"id":"242438","messageId":"20140521230139.GJ12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-35-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T23:01:39Z","receivedAt":"2014-05-21T23:01:39Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change prune_ref to delete the ref using a ref transaction. To do this we also\n> need to add a new flag REF_ISPRUNING that will tell the transaction that we\n> do not want to delete this ref from the packed refs.\n\nInteresting. Since the flag is per ref update, it even would allow\ndeleting some refs and pruning others in the same transaction. Makes\nsense.\n\nLooks like this doesn't batch up multiple ref-prunings into a single\ntransaction. Makes sense (it would just make the period while refs\nare locked longer without having any real benefit).\n\n[...]\n> +#define REF_ISPRUNING\t0x0100\n\nCan this conflict with bit values declared elsewhere some day? It\nwould be more comfortable if refs.h also had a note about bits >=\n0x100 being reserved for private use.\n\nThe rest of the patch looks good.\n\nThanks,\nJonathan\n"},{"id":"242440","messageId":"20140521232211.GK12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-36-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T23:22:11Z","receivedAt":"2014-05-21T23:22:11Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/refs.c\n> +++ b/refs.c\n[...]\n> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n> \n> int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n> {\n> -\tstruct ref_lock *lock;\n> -\tint ret = 0, flag = 0;\n> +\tstruct ref_transaction *transaction;\n> \n[...]\n> -\tlock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);\n> +\tif (!transaction ||\n> +\t ref_transaction_delete(transaction, refname, sha1, delopt,\n> +\t\t\t\t sha1 && !is_null_sha1(sha1), &err) ||\n\nWhat should happen when is_null_sha1(sha1)? In that case the\ncaller has asked us to check that the ref doesn't exist before\ndeleting it, which doesn't make a lot of sense, but the old\ncode did exactly that if I read it correctly. The new code\nseems to disable verification instead.\n\nDo we know that no callers call delete_ref with such an argument?\nWould it make sense to just die with a \"BUG:\" message to make\nthe contract more clear?\n\n[...]\n> -\tunlink_or_warn(git_path(\"logs/%s\", lock->ref_name));\n\nWhen does the reflog get deleted in the new code?\n\nThanks,\nJonathan\n"},{"id":"242441","messageId":"20140521234717.GL12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-37-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T23:47:17Z","receivedAt":"2014-05-21T23:47:17Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Change the reference transactions so that we pass the reflog message\n> through to the create/delete/update function instead of the commit message.\n\nNice.\n\n[...]\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -673,7 +673,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \t\t\t}\n> \t\t}\n> \t}\n> -\n> \tif (rc & STORE_REF_ERROR_DF_CONFLICT)\n> \t\terror(_(\"some local refs could not be updated; try running\\n\"\n> \t\t \" 'git remote prune %s' to remove any old, conflicting \"\n\nStray whitespace change?\n\n[...]\n> --- a/refs.c\n> +++ b/refs.c\n[...]\n> @@ -3264,6 +3264,7 @@ struct ref_update {\n> \tint have_old; /* 1 if old_sha1 is valid, 0 otherwise */\n> \tstruct ref_lock *lock;\n> \tint type;\n> +\tconst char *msg;\n> \tconst char refname[FLEX_ARRAY];\n\nShould be 'char *msg' since we own the memory (or perhaps a strbuf).\n\n[...]\n> @@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)\n> \tif (!transaction)\n> \t\treturn;\n> \n> -\tfor (i = 0; i < transaction->nr; i++)\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t free((char *)transaction->updates[i]->msg);\n> \t\tfree(transaction->updates[i]);\n\nWhitespace?\n\nNo need to cast.\n\nThe rest of the patch looks good.\n\nThanks,\nJonathan\n"},{"id":"242442","messageId":"20140521235040.GM12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-38-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 37/44] refs.c: pass NULL as *flags to read_ref_full","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T23:50:40Z","receivedAt":"2014-05-21T23:50:40Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n\nReviewed-by: Jonathan Nieder \n"},{"id":"242443","messageId":"20140521235741.GN12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-39-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-21T23:57:41Z","receivedAt":"2014-05-21T23:57:41Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This means that most loose refs will no longer be present after the rename\n\nIs this to handle the \"git branch -m foo/bar foo\" case or for some other\npurpose?\n\n[...]\n> --- a/t/t3200-branch.sh\n> +++ b/t/t3200-branch.sh\n> @@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '\n> \tgit symbolic-ref refs/heads/master2 refs/heads/master &&\n> \ttest_must_fail git branch -m master2 master3 &&\n> \tgit symbolic-ref refs/heads/master2 &&\n> -\ttest_path_is_file .git/refs/heads/master &&\n> +\ttest_path_is_missing .git/refs/heads/master &&\n> \ttest_path_is_missing .git/refs/heads/master3\n\nIt's kind of silly that this test is mucking about in the .git directory\nat all. Shouldn't the check be something like\n\n\tgit rev-parse --verify refs/heads/master &&\n\ttest_must_fail git symbolic-ref refs/heads/master3 &&\n\ttest_must_fail git rev-parse refs/heads/master3\n\n?\n"},{"id":"242449","messageId":"20140522014242.GO12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-40-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T01:42:42Z","receivedAt":"2014-05-22T01:42:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n> \tint missing = 0;\n> \tint attempts_remaining = 3;\n> \n> +\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n> +\t\treturn NULL;\n\nAre we sure that no callers are locking a poorly named ref\nas preparation for repairing it?\n\nTo see what works already in that vein, I tried:\n\n\t$ git rev-parse HEAD >.git/refs/heads/foo..bar\n\t$ git branch -m foo..bar something-saner\n\tfatal: Invalid branch name: 'foo..bar'\n\n\"git branch -m\" has an explicit codepath (\"recovery = 1;\") to handle\nthis case, but it looks like it was not well tested and regressed in\nv1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has\nthe right format, 2011-09-15).\n\nIs what the recovery codepath of branch -m does misguided? One\nschool of thought would be that people with malformed refs are\nresponsible for recovering using low-level tools like \"mv\" and \"vi\"\ninstead of normal git commands since normal git commands should never\nhave created such a bad situation. Another school of thought would\nassert that\n\n * git commands can have bugs\n * the format checked by check_refname_format() keeps getting stricter\n over time\n\nwhich means it's nice when people can recover with 'update-ref -d'\nor 'branch -m'. It's not obvious to me what the right thing to do\nhere is (maybe a special flag to be attached to a ref update during\nrecovery?).\n\nHope that helps,\nJonathan\n"},{"id":"242490","messageId":"CAL=YDWmwfdb7BEoyFV4SQjC+wySDWzE1E4S3hrhsVy3UZOa5Lw@mail.gmail.com","threadId":"36669","inReplyTo":"20140521232211.GK12314@google.com","subject":"Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T15:32:47Z","receivedAt":"2014-05-22T15:32:47Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 4:22 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/refs.c\n>> +++ b/refs.c\n> [...]\n>> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n>>\n>> int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n>> {\n>> - struct ref_lock *lock;\n>> - int ret = 0, flag = 0;\n>> + struct ref_transaction *transaction;\n>>\n> [...]\n>> - lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);\n>> + if (!transaction ||\n>> + ref_transaction_delete(transaction, refname, sha1, delopt,\n>> + sha1 && !is_null_sha1(sha1), &err) ||\n>\n> What should happen when is_null_sha1(sha1)? In that case the\n> caller has asked us to check that the ref doesn't exist before\n> deleting it, which doesn't make a lot of sense, but the old\n> code did exactly that if I read it correctly. The new code\n> seems to disable verification instead.\n>\n> Do we know that no callers call delete_ref with such an argument?\n> Would it make sense to just die with a \"BUG:\" message to make\n> the contract more clear?\n\nThere are no callers that pass in null_sha1 explicitely and all tests pass.\nI have changed it to a die(\"BUG... to make it more explicit as you suggested.\n\n>\n> [...]\n>> - unlink_or_warn(git_path(\"logs/%s\", lock->ref_name));\n>\n> When does the reflog get deleted in the new code?\n\nIt is deleted towards the end of ref_transaction_commit, after the ref\nitself has been deleted.\n\nThanks!\n\nronnie sahlberg\n"},{"id":"242491","messageId":"CAL=YDWkFC=67KjNSLKWLGHs2M33uVLWgzBy-LAPaUuTVZDgAnQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140521234717.GL12314@google.com","subject":"Re: [PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T15:40:22Z","receivedAt":"2014-05-22T15:40:22Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 4:47 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Change the reference transactions so that we pass the reflog message\n>> through to the create/delete/update function instead of the commit message.\n>\n> Nice.\n>\n> [...]\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -673,7 +673,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n>> }\n>> }\n>> }\n>> -\n>> if (rc & STORE_REF_ERROR_DF_CONFLICT)\n>> error(_(\"some local refs could not be updated; try running\\n\"\n>> \" 'git remote prune %s' to remove any old, conflicting \"\n>\n> Stray whitespace change?\n\nFixed.\n\n>\n> [...]\n>> --- a/refs.c\n>> +++ b/refs.c\n> [...]\n>> @@ -3264,6 +3264,7 @@ struct ref_update {\n>> int have_old; /* 1 if old_sha1 is valid, 0 otherwise */\n>> struct ref_lock *lock;\n>> int type;\n>> + const char *msg;\n>> const char refname[FLEX_ARRAY];\n>\n> Should be 'char *msg' since we own the memory (or perhaps a strbuf).\n\nI changed it to char *msg, but it still might/should be const. We own\nthe memory but we are not supposed to modify the content\n(after we have copied what the caller gave us).\n\nYes, we should change it into a strbuf at some stage.\n\n>\n> [...]\n>> @@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)\n>> if (!transaction)\n>> return;\n>>\n>> - for (i = 0; i < transaction->nr; i++)\n>> + for (i = 0; i < transaction->nr; i++) {\n>> + free((char *)transaction->updates[i]->msg);\n>> free(transaction->updates[i]);\n>\n> Whitespace?\n\nFixed.\n\n>\n> No need to cast.\n\nDone.\n\nThanks!\nronnie sahlberg\n"},{"id":"242492","messageId":"CAL=YDW=vY_09An0encywRoAexAEoQK=SbOe5Dt_FJmsFnwoCrg@mail.gmail.com","threadId":"36669","inReplyTo":"20140521235741.GN12314@google.com","subject":"Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T15:50:09Z","receivedAt":"2014-05-22T15:50:09Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> This means that most loose refs will no longer be present after the rename\n>\n> Is this to handle the \"git branch -m foo/bar foo\" case or for some other\n> purpose?\n\nYes. That is the main reason.\n\n>\n> [...]\n>> --- a/t/t3200-branch.sh\n>> +++ b/t/t3200-branch.sh\n>> @@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '\n>> git symbolic-ref refs/heads/master2 refs/heads/master &&\n>> test_must_fail git branch -m master2 master3 &&\n>> git symbolic-ref refs/heads/master2 &&\n>> - test_path_is_file .git/refs/heads/master &&\n>> + test_path_is_missing .git/refs/heads/master &&\n>> test_path_is_missing .git/refs/heads/master3\n>\n> It's kind of silly that this test is mucking about in the .git directory\n> at all. Shouldn't the check be something like\n>\n> git rev-parse --verify refs/heads/master &&\n> test_must_fail git symbolic-ref refs/heads/master3 &&\n> test_must_fail git rev-parse refs/heads/master3\n>\n> ?\n\nThanks. I updated the test with your change.\n\nThere is a whole bunch of tests that are like and access files directly..\nTesting if a path exists or not, or checking that a reflog file\ncontains x number of lines. etc.\n\nAll these tests will be updated to not access the files directly once\nI start mucking around with a TDB based refs backend.\n"},{"id":"242494","messageId":"CAL=YDWmPDFOgcFLDgogKQxqkP+D1bUsywWC+Xca1AxwA-8Y6hQ@mail.gmail.com","threadId":"36669","inReplyTo":"CAL=YDWmwfdb7BEoyFV4SQjC+wySDWzE1E4S3hrhsVy3UZOa5Lw@mail.gmail.com","subject":"Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T16:31:11Z","receivedAt":"2014-05-22T16:31:11Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 8:32 AM, Ronnie Sahlberg wrote:\n> On Wed, May 21, 2014 at 4:22 PM, Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n>>\n>>> --- a/refs.c\n>>> +++ b/refs.c\n>> [...]\n>>> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n>>>\n>>> int delete_ref(const char *refname, const unsigned char *sha1, int delopt)\n>>> {\n>>> - struct ref_lock *lock;\n>>> - int ret = 0, flag = 0;\n>>> + struct ref_transaction *transaction;\n>>>\n>> [...]\n>>> - lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);\n>>> + if (!transaction ||\n>>> + ref_transaction_delete(transaction, refname, sha1, delopt,\n>>> + sha1 && !is_null_sha1(sha1), &err) ||\n>>\n>> What should happen when is_null_sha1(sha1)? In that case the\n>> caller has asked us to check that the ref doesn't exist before\n>> deleting it, which doesn't make a lot of sense, but the old\n>> code did exactly that if I read it correctly. The new code\n>> seems to disable verification instead.\n>>\n>> Do we know that no callers call delete_ref with such an argument?\n>> Would it make sense to just die with a \"BUG:\" message to make\n>> the contract more clear?\n>\n> There are no callers that pass in null_sha1 explicitely and all tests pass.\n> I have changed it to a die(\"BUG... to make it more explicit as you suggested.\n\nStrike that.\n\nWhile there are no callers I can see that deliberately send null_sha1\nit can happen because resolve_ref_unsafe(reading=0) calls\nhandle_missing_loose_ref which will clear sha1\nif the ref is missing.\n\nThis can happen for either symrefs or refs that are soft links that\npoint to a nonexisting ref.\n\n\n\n>\n>>\n>> [...]\n>>> - unlink_or_warn(git_path(\"logs/%s\", lock->ref_name));\n>>\n>> When does the reflog get deleted in the new code?\n>\n> It is deleted towards the end of ref_transaction_commit, after the ref\n> itself has been deleted.\n>\n> Thanks!\n>\n> ronnie sahlberg\n"},{"id":"242498","messageId":"CAL=YDW=g=5ux4tThcK3EEmPuzqmwGBX+7au4MMWnfL_NLq897A@mail.gmail.com","threadId":"36669","inReplyTo":"20140521220710.GG12314@google.com","subject":"Re: [PATCH v8 32/44] refs.c: remove the update_ref_write function","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T16:49:43Z","receivedAt":"2014-05-22T16:49:43Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 3:07 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> +++ b/refs.c\n> [...]\n>> @@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n>> struct ref_update *update = updates[i];\n>>\n>> if (!is_null_sha1(update->new_sha1)) {\n>> - ret = update_ref_write(msg,\n>> - update->refname,\n>> - update->new_sha1,\n>> - update->lock, err,\n>> - UPDATE_REFS_QUIET_ON_ERR);\n>> + ret = write_ref_sha1(update->lock, update->new_sha1,\n>> + msg);\n>\n> This changes the return value on error from 1 to -1. That seems like a\n> good change. It's probably worth mentioning in the commit message.\n\nDone.\n"},{"id":"242499","messageId":"CAL=YDWkGWwBaNqACS=wYj-kFg2FLddp8n4U=LrUdR0-4sDh9_g@mail.gmail.com","threadId":"36669","inReplyTo":"20140521230139.GJ12314@google.com","subject":"Re: [PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T16:56:36Z","receivedAt":"2014-05-22T16:56:36Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"Added a comment that any flags >=0x100 are reserved for internal use.\n\n\nOn Wed, May 21, 2014 at 4:01 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Change prune_ref to delete the ref using a ref transaction. To do this we also\n>> need to add a new flag REF_ISPRUNING that will tell the transaction that we\n>> do not want to delete this ref from the packed refs.\n>\n> Interesting. Since the flag is per ref update, it even would allow\n> deleting some refs and pruning others in the same transaction. Makes\n> sense.\n>\n> Looks like this doesn't batch up multiple ref-prunings into a single\n> transaction. Makes sense (it would just make the period while refs\n> are locked longer without having any real benefit).\n>\n> [...]\n>> +#define REF_ISPRUNING 0x0100\n>\n> Can this conflict with bit values declared elsewhere some day? It\n> would be more comfortable if refs.h also had a note about bits >=\n> 0x100 being reserved for private use.\n>\n> The rest of the patch looks good.\n>\n> Thanks,\n> Jonathan\n"},{"id":"242500","messageId":"CAL=YDW=5FozayJY2=_+eBn_fUsNQZuHhDKaSvzTiHOMJPA_JMw@mail.gmail.com","threadId":"36669","inReplyTo":"20140521222201.GI12314@google.com","subject":"Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T17:15:16Z","receivedAt":"2014-05-22T17:15:16Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 3:22 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> Please pull my ref-transactions branch.\n>\n> I'm at bd5736cb (2014-05-21 13:46) now.\n>\n>> On Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder wrote:\n>>> Ronnie Sahlberg wrote:\n>\n>>>> --- a/refs.c\n>>>> +++ b/refs.c\n>>>> @@ -3308,6 +3308,12 @@ struct ref_update {\n>>>> const char refname[FLEX_ARRAY];\n>>>> };\n>>>>\n>>>> +enum ref_transaction_status {\n>>>> + REF_TRANSACTION_OPEN = 0,\n>>>> + REF_TRANSACTION_CLOSED = 1,\n>>>> + REF_TRANSACTION_ERROR = 2,\n>>>\n>>> What is the difference between _TRANSACTION_CLOSED and\n>>> _TRANSACTION_ERROR?\n>>\n>> Closed is a transaction that has been committed successfully, and\n>> which we can not do any more updates onto.\n>> Error is a transaction that has failed, and which we can not do any\n>> more updates onto.\n>\n> That means that both mean the caller cannot do any more updates,\n> right? Why not call them both _CLOSED?\n>\n>> The distinction could be useful if in the future we add support to\n>> reuse a transaction\n>\n> If the distinction becomes useful in the future, wouldn't that\n> be the moment to add a new state?\n>\n> I don't mean that there has to be a big useful distinction. E.g.,\n> maybe the idea is that when using a debugger it can be useful to see\n> the difference between _ERROR and _CLOSED or something, and I think\n> that would be fine as long as the intended meaning is documented\n> (e.g., using comments). My only complaint is that it's hard to\n> understand the code and keep track of which status should be used in a\n> given place unless there is some distinction between them.\n\nI have documented the transaction states in refs.c\n\nThanks.\nronnie sahlberg\n"},{"id":"242501","messageId":"CAL=YDW=fqDFFMomX-GDyANsABz6E5zww+JKgD6V0A+mc6Ja96g@mail.gmail.com","threadId":"36669","inReplyTo":"20140522014242.GO12314@google.com","subject":"Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T17:28:47Z","receivedAt":"2014-05-22T17:28:47Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n>> int missing = 0;\n>> int attempts_remaining = 3;\n>>\n>> + if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))\n>> + return NULL;\n>\n> Are we sure that no callers are locking a poorly named ref\n> as preparation for repairing it?\n>\n> To see what works already in that vein, I tried:\n>\n> $ git rev-parse HEAD >.git/refs/heads/foo..bar\n> $ git branch -m foo..bar something-saner\n> fatal: Invalid branch name: 'foo..bar'\n>\n> \"git branch -m\" has an explicit codepath (\"recovery = 1;\") to handle\n> this case, but it looks like it was not well tested and regressed in\n> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has\n> the right format, 2011-09-15).\n>\n> Is what the recovery codepath of branch -m does misguided? One\n> school of thought would be that people with malformed refs are\n> responsible for recovering using low-level tools like \"mv\" and \"vi\"\n> instead of normal git commands since normal git commands should never\n> have created such a bad situation. Another school of thought would\n> assert that\n>\n> * git commands can have bugs\n> * the format checked by check_refname_format() keeps getting stricter\n> over time\n>\n> which means it's nice when people can recover with 'update-ref -d'\n> or 'branch -m'. It's not obvious to me what the right thing to do\n> here is (maybe a special flag to be attached to a ref update during\n> recovery?).\n>\n\nI don't think we should allow external caller any other way to\naccess/modify refs than through\nthe transaction api. This may make it awkward to handle cases such as\nfoo/../bar was created and how would you now delete it ?\n\nThe exception to this I think would be 'git fsck'. We could allow fsck\nto have low level access to the backing store and allow it to access\nthe files directly,\nor allow fsck to set magic flags that disable various checks.\n\n\n\n\n> Hope that helps,\n> Jonathan\n"},{"id":"242502","messageId":"20140522174427.GP12314@google.com","threadId":"36669","inReplyTo":"CAL=YDW=fqDFFMomX-GDyANsABz6E5zww+JKgD6V0A+mc6Ja96g@mail.gmail.com","subject":"Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T17:44:27Z","receivedAt":"2014-05-22T17:44:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n> On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder wrote:\n\n>> $ git rev-parse HEAD >.git/refs/heads/foo..bar\n>> $ git branch -m foo..bar something-saner\n>> fatal: Invalid branch name: 'foo..bar'\n>>\n>> \"git branch -m\" has an explicit codepath (\"recovery = 1;\") to handle\n>> this case, but it looks like it was not well tested and regressed in\n>> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has\n>> the right format, 2011-09-15).\n>>\n>> Is what the recovery codepath of branch -m does misguided?\n[...]\n> I don't think we should allow external caller any other way to\n> access/modify refs than through\n> the transaction api. This may make it awkward to handle cases such as\n> foo/../bar was created and how would you now delete it ?\n>\n> The exception to this I think would be 'git fsck'. We could allow fsck\n> to have low level access to the backing store and allow it to access\n> the files directly,\n> or allow fsck to set magic flags that disable various checks.\n\nInteresting. Do you mean that 'git fsck' should notice invalid refnames\nand have an option to repair them on its own?\n\nThat would be a big change from what git fsck currently does --- it's\ncurrently read-only (except for \"fsck --lost-found\", which is read-only\nexcept in the .git/lost-found/ directory). If there's an option to\ncheck refnames only and not have to wait for the full check of all\nobjects then I think it makes sense.\n\nIn the meantime, 'git branch -m' and 'git update-ref -d' (which were\nthe historical ways to do this kind of repair) are already broken, so\nat least this patch doesn't seem to make anything worse. :/ (I\nhaven't checked all callers, though.)\n\nJonathan\n"},{"id":"242503","messageId":"20140522175153.GQ12314@google.com","threadId":"36669","inReplyTo":"CAL=YDW=vY_09An0encywRoAexAEoQK=SbOe5Dt_FJmsFnwoCrg@mail.gmail.com","subject":"Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T17:51:53Z","receivedAt":"2014-05-22T17:51:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n> On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n\n>>> This means that most loose refs will no longer be present after the rename\n>>\n>> Is this to handle the \"git branch -m foo/bar foo\" case or for some other\n>> purpose?\n>\n> Yes. That is the main reason.\n\nThanks. Please say so in the log message. Remember that people of\nthe future debugging and trying to figure out how to fix the latest\nregression without breaking something else and whether to just revert\nyour patch cannot read your mind. ;-)\n\nHope that helps,\nJonathan\n"},{"id":"242504","messageId":"20140522175354.GR12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-41-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 40/44] refs.c: call lock_ref_sha1_basic directly from commit","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T17:53:54Z","receivedAt":"2014-05-22T17:53:54Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs.c | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n\nReviewed-by: Jonathan Nieder \n"},{"id":"242505","messageId":"CAL=YDWk5EbkLy7Jd2Mv6-rRKry=6dqgoSqjFj0ZHaWw4xR-_MA@mail.gmail.com","threadId":"36669","inReplyTo":"20140522174427.GP12314@google.com","subject":"Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T17:57:23Z","receivedAt":"2014-05-22T17:57:23Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 10:44 AM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>> On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder wrote:\n>\n>>> $ git rev-parse HEAD >.git/refs/heads/foo..bar\n>>> $ git branch -m foo..bar something-saner\n>>> fatal: Invalid branch name: 'foo..bar'\n>>>\n>>> \"git branch -m\" has an explicit codepath (\"recovery = 1;\") to handle\n>>> this case, but it looks like it was not well tested and regressed in\n>>> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has\n>>> the right format, 2011-09-15).\n>>>\n>>> Is what the recovery codepath of branch -m does misguided?\n> [...]\n>> I don't think we should allow external caller any other way to\n>> access/modify refs than through\n>> the transaction api. This may make it awkward to handle cases such as\n>> foo/../bar was created and how would you now delete it ?\n>>\n>> The exception to this I think would be 'git fsck'. We could allow fsck\n>> to have low level access to the backing store and allow it to access\n>> the files directly,\n>> or allow fsck to set magic flags that disable various checks.\n>\n> Interesting. Do you mean that 'git fsck' should notice invalid refnames\n> and have an option to repair them on its own?\n\nI think that would be useful.\nBut it would prompt the user before it does any changes. Even for a\nfsck it would be rude to do any changes without asking the user.\n\n'.git/refs/heads/foo^G^G^G.........bar' it not a valid ref.\n[s kip] [d elete] [r ename] ?\n\n\n>\n> That would be a big change from what git fsck currently does --- it's\n> currently read-only (except for \"fsck --lost-found\", which is read-only\n> except in the .git/lost-found/ directory). If there's an option to\n> check refnames only and not have to wait for the full check of all\n> objects then I think it makes sense.\n>\n> In the meantime, 'git branch -m' and 'git update-ref -d' (which were\n> the historical ways to do this kind of repair) are already broken, so\n> at least this patch doesn't seem to make anything worse. :/ (I\n> haven't checked all callers, though.)\n>\n> Jonathan\n"},{"id":"242506","messageId":"CAL=YDWn8qh7SMHR+T1rsxB8u=Qd6MOSTc9Wo8pUc-jy5tNDdjg@mail.gmail.com","threadId":"36669","inReplyTo":"20140522175153.GQ12314@google.com","subject":"Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T18:02:27Z","receivedAt":"2014-05-22T18:02:27Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 10:51 AM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>> On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder wrote:\n>>> Ronnie Sahlberg wrote:\n>\n>>>> This means that most loose refs will no longer be present after the rename\n>>>\n>>> Is this to handle the \"git branch -m foo/bar foo\" case or for some other\n>>> purpose?\n>>\n>> Yes. That is the main reason.\n>\n> Thanks. Please say so in the log message. Remember that people of\n> the future debugging and trying to figure out how to fix the latest\n> regression without breaking something else and whether to just revert\n> your patch cannot read your mind. ;-)\n>\n\nDone.\n"},{"id":"242508","messageId":"20140522181722.GT12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-42-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T18:17:22Z","receivedAt":"2014-05-22T18:17:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nRonnie Sahlberg wrote:\n\n> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.\n> This flag indicates that the ref does not exist as a loose ref andf only as\n> a packed ref. If this is the case we then change the commit code so that\n> we skip taking out a lock file and we skip calling delete_ref_loose.\n\nThis seems wrong. Can't someone else create a loose ref which will\nshadow the packed ref and break the serializability of updates to this\nref?\n\nThe above doesn't explain why we want to avoid locking the loose ref,\nbut I assume it's for the sake of the \"git branch -m foo/bar foo\"\ncase. For that case, wouldn't the following sequence of filesystem\noperations work?\n\n\t- create $GIT_DIR/refs/heads/foo/bar.lock\n\t- create $GIT_DIR/refs/heads/foo.lock\n\t- if $GIT_DIR/refs/heads/foo/bar exists, add the ref to\n\t packed-refs (using the usual lockfile-update mechanism)\n\t and unlink $GIT_DIR/refs/heads/foo/bar\n\t- verify that current foo and foo/bar state are okay. If\n\t not, roll back.\n\t- unlink $GIT_DIR/refs/heads/foo/bar.lock\n\t- rmdir $GIT_DIR/refs/heads/foo\n\t- rename $GIT_DIR/refs/heads/foo.lock into place\n\nOr:\n\n\t- create $GIT_DIR/refs/heads/foo/bar.lock\n\t- create $GIT_DIR/refs/heads/foo.lock\n\t- verify state of all relevant refs\n\n\t- update packed-refs to remove refs/heads/foo/bar and\n\t add refs/heads/foo\n\n\t- unlink $GIT_DIR/refs/heads/foo/bar\n\t- unlink $GIT_DIR/refs/heads/foo\n\t- unlink $GIT_DIR/refs/heads/foo/bar.lock\n\t- unlink $GIT_DIR/refs/heads/foo.lock\n\nThanks,\nJonathan\n"},{"id":"242517","messageId":"CAL=YDWkUa2Ut=1iaMXgnjgte6g5jvXR49LBiEUNEkwv4Z-wO_w@mail.gmail.com","threadId":"36669","inReplyTo":"20140522181722.GT12314@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T19:12:42Z","receivedAt":"2014-05-22T19:12:42Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 11:17 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Ronnie Sahlberg wrote:\n>\n>> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.\n>> This flag indicates that the ref does not exist as a loose ref andf only as\n>> a packed ref. If this is the case we then change the commit code so that\n>> we skip taking out a lock file and we skip calling delete_ref_loose.\n>\n> This seems wrong. Can't someone else create a loose ref which will\n> shadow the packed ref and break the serializability of updates to this\n> ref?\n>\n> The above doesn't explain why we want to avoid locking the loose ref,\n> but I assume it's for the sake of the \"git branch -m foo/bar foo\"\n> case. For that case, wouldn't the following sequence of filesystem\n> operations work?\n>\n> - create $GIT_DIR/refs/heads/foo/bar.lock\n> - create $GIT_DIR/refs/heads/foo.lock\n> - if $GIT_DIR/refs/heads/foo/bar exists, add the ref to\n> packed-refs (using the usual lockfile-update mechanism)\n> and unlink $GIT_DIR/refs/heads/foo/bar\n> - verify that current foo and foo/bar state are okay. If\n> not, roll back.\n> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n> - rmdir $GIT_DIR/refs/heads/foo\n> - rename $GIT_DIR/refs/heads/foo.lock into place\n>\n> Or:\n>\n> - create $GIT_DIR/refs/heads/foo/bar.lock\n> - create $GIT_DIR/refs/heads/foo.lock\n> - verify state of all relevant refs\n>\n> - update packed-refs to remove refs/heads/foo/bar and\n> add refs/heads/foo\n>\n> - unlink $GIT_DIR/refs/heads/foo/bar\n> - unlink $GIT_DIR/refs/heads/foo\n> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n> - unlink $GIT_DIR/refs/heads/foo.lock\n>\n\n\nI removed all the rename_ref related patches for now. rename_ref is\nprobably best implemented specifically for each backend anyway.\n\nI will still produce a separate patch that will do something like this\nyou suggested\n(since rename_ref is still racy and fragile)\n\n> - create $GIT_DIR/refs/heads/foo/bar.lock\n> - create $GIT_DIR/refs/heads/foo.lock\n> - verify state of all relevant refs\n>\n> - update packed-refs to remove refs/heads/foo/bar and\n> add refs/heads/foo\n>\n> - unlink $GIT_DIR/refs/heads/foo/bar\n> - unlink $GIT_DIR/refs/heads/foo\n> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n> - unlink $GIT_DIR/refs/heads/foo.lock\n>\n\nThanks\nronnie sahlberg\n"},{"id":"242519","messageId":"20140522192717.GU12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-43-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T19:27:17Z","receivedAt":"2014-05-22T19:27:17Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -798,11 +798,19 @@ struct name_conflict_cb {\n> \tconst char *refname;\n> \tconst char *oldrefname;\n> \tconst char *conflicting_refname;\n> +\tconst char **skip;\n> +\tint skipnum;\n\nWould a struct string_list make sense here? (See\nDocumentation/technical/api-string-list.txt.)\n\n[...]\n> };\n> \n> static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n> {\n> \tstruct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;\n> +\tint i;\n> +\tfor(i = 0; i < data->skipnum; i++) {\n\n(style nit) missing space after 'for'.\n\n> +\t\tif (!strcmp(entry->name, data->skip[i])) {\n> +\t\t\treturn 0;\n> +\t\t}\n\nStyle: git tends to avoid braces around a single-line if/for/etc body.\n\n[...]\n> @@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n> * conflicting with the name of an existing reference in dir. If\n> * oldrefname is non-NULL, ignore potential conflicts with oldrefname\n> * (e.g., because oldrefname is scheduled for deletion in the same\n> - * operation).\n> + * operation). skip contains a list of refs we want to skip checking for\n> + * conflicts with. Refs may be skipped due to us knowing that it will\n> + * be deleted later during a transaction that deletes one reference and then\n> + * creates a new conflicting reference. For example a rename from m to m/m.\n\nThis example of \"Refs may be skipped due to\" seems overly complicated.\nIsn't the idea just that skip contains a list of refs scheduled for\ndeletion in this transaction, since they shouldn't be treated as\nconflicts at all (for example when renamining m to m/m)?\n\nI wonder if there's some way to make use of the result of the naive\nrefname_available check to decide what to do when creating a ref.\n\nE.g.: if a refname would be available except there's a ref being\ndeleted in the way, we could do one of the following:\n\n a. delete all relevant loose refs and perform the transaction in\n packed-refs, or\n\n b. order operations to avoid the D/F conflict, even with loose refs\n (the hardest case is if the ref being deleted uses a directory\n and we want to create a file with the same name. But that's\n still doable if we're willing to rmdir when needed as part of\n the loop to commit changes)\n\nThe packed-refs trick (a) seems much simpler, but either should work.\n\nThis could be done e.g. by checking is_refname_available with an empty\nlist first before doing the real thing with a list of exclusions.\n\n[...]\n> @@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n> \tint log = !lstat(git_path(\"logs/%s\", oldrefname), &loginfo);\n> \tconst char *symref = NULL;\n> \n> +\tif (!strcmp(oldrefname, newrefname))\n> +\t\treturn 0;\n\nWhat is the intended result if I try to rename a nonexistent ref or an\nexistent symref to its own name?\n\nSorry to be so fussy about this part. It's not that I think that this\nchange is trying to do something bad --- in fact, it's more the\nopposite, that I'm excited to see git learning to have a better\nunderstanding and handling of refname D/F conflicts.\n\nThat would allow:\n\n * \"git fetch --prune\" working as a single transaction even if\n the repository being fetched from removed a refs/heads/topic\n branch and created refs/heads/topic/1 and refs/heads/topic/2\n\n * \"git fast-import\" and \"git fetch --mirror\" learning the same trick\n\n * fewer code paths having to be touched to be able to (optionally)\n let git actually tolerate D/F conflicts, for people who want to\n have 'topic', 'topic/1', and 'topic/2' branches at the same time.\n This could be turned on by default for remote-tracking refs. It\n would be especially nice for people on Windows and Mac OS where\n there can be D/F conflicts that people on Linux didn't notice due\n to case-sensitivity.\n\n Longer term, through a configuration that starts turned off by\n default and has the default flipped as more people have upgraded\n git, this could make D/F conflicts in refnames stop being an error\n altogether.\n\nSo it's kind of exciting to see, even though it's fussy to get it\nright.\n\nThanks,\nJonathan\n"},{"id":"242522","messageId":"20140522195152.GW12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T19:51:52Z","receivedAt":"2014-05-22T19:51:52Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This version completes the work to convert all ref updates to use transactions.\n\nFinally got through this. It had thorny bits but generally goes in a\nvery good direction. Thanks for a pleasant read.\n\nFeel free to send another iteration if you'd like review for the newer\ncode. I expect except for the part about renames that most of what's\nleft is just nits so it should go faster.\n\nThanks,\nJonathan\n"},{"id":"242523","messageId":"20140522195808.GX12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T19:58:08Z","receivedAt":"2014-05-22T19:58:08Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This patch series can also be found at\n> https://github.com/rsahlberg/git/tree/ref-transactions\n\nThoughts on 65a1cb7b (2014-05-22 12:08):\n\n 01/40 remove ref_transaction_rollback\n Reviewed-by: Jonathan Nieder \n\n 02/40 constify the sha arguments for ref_transaction_create|delete|update\n still Reviewed-by: Jonathan Nieder \n\n 03/40 allow passing NULL to ref_transaction_free\n The third paragraph (\"This allows us to write code like\") is still\n confusing. Why not drop that paragraph?\n\n 04/40 --- oh, lunch time. Will pick up when I get back.\n\nThanks,\nJonathan\n"},{"id":"242531","messageId":"20140522220851.GY12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T22:08:51Z","receivedAt":"2014-05-22T22:08:51Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This patch series can also be found at\n> https://github.com/rsahlberg/git/tree/ref-transactions\n\nThoughts on 65a1cb7b (2014-05-22 12:08):\n\n 04/40 add a strbuf argument to ref_transaction_commit for error logging\n\n Ideally this would come after the functions it calls so the comment\n \"If error is non-NULL we will add an error string to it to explain\n why the transaction failed\" would be true already. But rearranging\n the series for that would be more fuss than it's worth IMHO.\n\n The sanity check\n\n\tint ref_transaction_commit(...)\n\t{\n\t\tint ret = ref_transaction_commit_real(...);\n\t\tif (ret && err && !err->len)\n\t\t\tdie(\"BUG: ref_transaction_commit forgot to write an error message\");\n\t\treturn ret;\n\t}\n\n shows no instances of forgotten error messages in cases exercised by\n tests, at least. And it's a step in the right direction.\n\n Reviewed-by: Jonathan Nieder \n\n 05/40 add an err argument to repack_without_refs\n unable_to_lock_strbuf could be called unable_to_lock_message (which\n would make its behavior more obvious imho).\n\n This makes errno meaningful when commit_packed_refs returns an error.\n Probably its API documentation should say so to make it obvious to\n people modifying it in the future to preserve that property or audit\n callers.\n\n Via the new call to unable_to_lock_..., repack_without_refs cares\n about errno after a failed call to lock_packed_refs. lock_packed_refs\n can only fail in hold_lock_file_for_update. hold_lock_file_for_update\n is a thin wrapper around lockfile.c::lock_file. lock_file can error\n out because\n\n\tstrlen(path) >= max_path_len\n\tadjust_shared_perm failed [calls error(), clobbers errno]\n\topen failed\n\n So lock_file needs a similar kind of fix, and it's probably worth\n updating API documentation for these calls to make it clear that\n their errno is used (though that's not a new problem since\n repack_without_refs already called unable_to_lock_error). Could be\n a separate, earlier patch since it's fixing an existing bug.\n\n 06/40 make ref_update_reject_duplicates take a strbuf argument for errors\n still Reviewed-by: Jonathan Nieder \n\n 07/40 add an err argument to delete_ref_loose\n The new unlink_or_err has an odd contract when the err argument is passed.\n On error:\n\n * if errno != ENOENT, it will append a message to err and return -1 (good)\n * if errno == ENOENT, it will not append a message to err but will\n still return -1.\n\n Perhaps it should return 0 when errno == ENOENT. After all, in that\n case the file does not exist any more, which is all we wanted. And it\n would save the caller from having to inspect errno.\n\n On failure we seem to add our own message to err, too, so the resulting\n message would be something like\n\n\tfatal: unable to unlink .git/refs/heads/master: \\\n\tPermission deniedfailed to delete loose ref '.git/refs/heads/master.lock'\n\n The second strbuf_addf is probably not needed.\n\n 08/40 make update_ref_write update a strbuf on failure\n still Reviewed-by: Jonathan Nieder \n \n 09/40 log transaction error from the update_ref\n No actual functional change intended, right? I'd say something like\n \"update-ref: use err argument to get error from ref_transaction_commit\"\n or something similar to make it clearer that this is just about\n changing APIs. Or if there's an intended functional change, then the\n commit message could say something about that.\n\n 10/40 remove the onerr argument to ref_transaction_commit\n still Reviewed-by: Jonathan Nieder \n"},{"id":"242537","messageId":"CAL=YDW=hytDiz5qzAMyBgXUgza+AcDhk_y3m3kAUmOdBG=F=vA@mail.gmail.com","threadId":"36669","inReplyTo":"CAL=YDWkUa2Ut=1iaMXgnjgte6g5jvXR49LBiEUNEkwv4Z-wO_w@mail.gmail.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-22T22:53:25Z","receivedAt":"2014-05-22T22:53:25Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"I hate rename_ref :-)\n\n\nI have reworked the transaction code to special case the deletion of\nthe old ref for n/n -> n and n -> n/n renames\nso that we can carefully avoid n/n.lock files to exist or prevent the\ndirectory <-> file transition for n during these renames.\n\nThis should allow us to have rename_ref becoming reasonably\nimplementation agnostic, aside for that it wants to lstat the ref to\nsee if it is a soft link, and re-use the code for other refs backends.\n\n\nPlease see the ref-transaction branch.\n\n\nOn Thu, May 22, 2014 at 12:12 PM, Ronnie Sahlberg wrote:\n> On Thu, May 22, 2014 at 11:17 AM, Jonathan Nieder wrote:\n>> Hi,\n>>\n>> Ronnie Sahlberg wrote:\n>>\n>>> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.\n>>> This flag indicates that the ref does not exist as a loose ref andf only as\n>>> a packed ref. If this is the case we then change the commit code so that\n>>> we skip taking out a lock file and we skip calling delete_ref_loose.\n>>\n>> This seems wrong. Can't someone else create a loose ref which will\n>> shadow the packed ref and break the serializability of updates to this\n>> ref?\n>>\n>> The above doesn't explain why we want to avoid locking the loose ref,\n>> but I assume it's for the sake of the \"git branch -m foo/bar foo\"\n>> case. For that case, wouldn't the following sequence of filesystem\n>> operations work?\n>>\n>> - create $GIT_DIR/refs/heads/foo/bar.lock\n>> - create $GIT_DIR/refs/heads/foo.lock\n>> - if $GIT_DIR/refs/heads/foo/bar exists, add the ref to\n>> packed-refs (using the usual lockfile-update mechanism)\n>> and unlink $GIT_DIR/refs/heads/foo/bar\n>> - verify that current foo and foo/bar state are okay. If\n>> not, roll back.\n>> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n>> - rmdir $GIT_DIR/refs/heads/foo\n>> - rename $GIT_DIR/refs/heads/foo.lock into place\n>>\n>> Or:\n>>\n>> - create $GIT_DIR/refs/heads/foo/bar.lock\n>> - create $GIT_DIR/refs/heads/foo.lock\n>> - verify state of all relevant refs\n>>\n>> - update packed-refs to remove refs/heads/foo/bar and\n>> add refs/heads/foo\n>>\n>> - unlink $GIT_DIR/refs/heads/foo/bar\n>> - unlink $GIT_DIR/refs/heads/foo\n>> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n>> - unlink $GIT_DIR/refs/heads/foo.lock\n>>\n>\n>\n> I removed all the rename_ref related patches for now. rename_ref is\n> probably best implemented specifically for each backend anyway.\n>\n> I will still produce a separate patch that will do something like this\n> you suggested\n> (since rename_ref is still racy and fragile)\n>\n>> - create $GIT_DIR/refs/heads/foo/bar.lock\n>> - create $GIT_DIR/refs/heads/foo.lock\n>> - verify state of all relevant refs\n>>\n>> - update packed-refs to remove refs/heads/foo/bar and\n>> add refs/heads/foo\n>>\n>> - unlink $GIT_DIR/refs/heads/foo/bar\n>> - unlink $GIT_DIR/refs/heads/foo\n>> - unlink $GIT_DIR/refs/heads/foo/bar.lock\n>> - unlink $GIT_DIR/refs/heads/foo.lock\n>>\n>\n> Thanks\n> ronnie sahlberg\n"},{"id":"242540","messageId":"20140522230802.GZ12314@google.com","threadId":"36669","inReplyTo":"1400174999-26786-1-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T23:08:02Z","receivedAt":"2014-05-22T23:08:02Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> This patch series can also be found at\n> https://github.com/rsahlberg/git/tree/ref-transactions\n\nContinuing with the review of 65a1cb7b (2014-05-22 12:08):\n\n 11/40 change ref_transaction_update() to do error checking and return status\n The \"there will be in the future\" sounds ominous. Do you have an\n example in mind? E.g., I suppose it would be nice if _update could\n notice D/F conflicts or connection to a database server closing early,\n but it's not clear to me whether the kind of errors you're talking\n about are that or something else.\n\n With or without such a clarification,\n Reviewed-by: Jonathan Nieder \n\n 12/40 change ref_transaction_create to do error checking and return status\n What does \"On failure the err buffer will be updated\" mean? Will\n it clear err and replace it with a message, append to err, or\n something else? Does the message explain the context or is the\n caller responsible for adding to it? Does the message end with a\n newline or is the caller responsible for adding one when printing it\n out?\n\n For cases like this where lots of functions have a similar API,\n API comments start to become potentially repetitive. It might be\n better to explain conventions at the top of the file or in\n Documentation/technical/api-refs.txt and say \"See the top of the\n file for error handling conventions\" or \"Returns non-zero and\n appends a message to err on error. See\n Documentation/technical/api-refs.txt for more details on error\n handling\".\n\n 13/40 ref_transaction_delete to check for error and return status\n Each successive commit dropped something from its subject. :)\n (First the (), then the verb.)\n\n Same comments as before about an example being useful for the\n log message and the API documentation on error handling being a\n bit vague.\n\n 14/40 make ref_transaction_begin take an err argument\n I found the \"failed to connect to mysql\" example instructive while\n doing reviews. Perhaps it would be worth mentioning in the commit\n message.\n\n Reviewed-by: Jonathan Nieder \n\n 15/40 add transaction.status and track OPEN/CLOSED/ERROR\n It says an ERRORed transaction cannot be committed and can be rolled\n back by calling _free. Can a CLOSED transaction be committed or\n _freed?\n\n What does \"faild\" mean in the documentation comments? (Maybe\n \"non-OPEN\"?)\n\n In the previous version of this patch passing a non-OPEN transaction\n would die(\"BUG: \"...) to diagnose the caller's mistake. Now I'm\n confused about the API: it seems you're allowed to pass a non-OPEN\n transaction but it doesn't append a message to 'err' in that case.\n Is this meant as a way to save the caller some typing, like\n fwrite/fclose do? (I've found people often make mistakes with the\n fwrite API fwiw but can understand the appeal of it.)\n\n Maybe with more context I'd like this. As is, it feels like a step\n in the wrong direction.\n\n 16/40 tag: use ref transactions when doing updates\n Reviewed-by: Jonathan Nieder \n\n 17/40 replace: use ref transactions when doing updates\n Reviewed-by: Jonathan Nieder \n\n 18/40 commit: use ref transactions for updates\n Reviewed-by: Jonathan Nieder \n \n 19/40 sequencer: use ref transactions for all ref updates\n This would be a lot simpler if the \"ref_transaction_commit should not\n free the transaction\" patch came before it (yes, sorry, killing the\n fun). I can push the result of a rebase doing that somewhere if you\n like.\n\n 20/40 fast-import: change update_branch to use ref transactions\n Likewise.\n"},{"id":"242546","messageId":"20140522234440.GA12314@google.com","threadId":"36669","inReplyTo":"CAL=YDW=hytDiz5qzAMyBgXUgza+AcDhk_y3m3kAUmOdBG=F=vA@mail.gmail.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T23:44:40Z","receivedAt":"2014-05-22T23:44:40Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n\n> I hate rename_ref :-)\n>\n> I have reworked the transaction code to special case the deletion of\n> the old ref for n/n -> n and n -> n/n renames\n> so that we can carefully avoid n/n.lock files to exist or prevent the\n> directory <-> file transition for n during these renames.\n\nThanks.\n\nI suspect the REF_ISRENAME flag shouldn't be needed. Wouldn't\nsomething like the following work (in _commit)?\n\n\tAllocate work space\n\tCopy sort, and reject duplicate refs\n\tAcquire all locks while verifying old values\n\t\tThis calls is_refname_available.\n\t\tIf a refname is unavailable, goto slowpath.\n\tPerform updates first so live commits remain referenced.\n\tPerform deletes now that updates are safely completed.\n\tDone.\n\n slowpath:\n\tAcquire locks, telling is_refname_available not to worry\n\t\tabout deleted refs.\n\tLock packed-refs.\n\tAdd all relevant refs to packed-refs (pack_if_possible_fn).\n\tCommit packed-refs.\n\tUnlink the corresponding loose refs so packed-refs\n\t\tbecomes authoritative for them.\n\tLock packed-refs.\n\tPerform updates and removals in the packed-refs cache.\n\tCommit packed-refs.\n\tRelease locks.\n\tDone.\n\nThis wouldn't be any slower for the case without D/F conflicts, and in\nthe D/F conflict case, it should work for arbitrary transactions\nthat want to remove one ref to make room for another.\n"},{"id":"242547","messageId":"20140522235351.GB12314@google.com","threadId":"36669","inReplyTo":"20140522234440.GA12314@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-22T23:53:51Z","receivedAt":"2014-05-22T23:53:51Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n\n>> I hate rename_ref :-)\n>>\n>> I have reworked the transaction code to special case the deletion of\n>> the old ref for n/n -> n and n -> n/n renames\n>> so that we can carefully avoid n/n.lock files to exist or prevent the\n>> directory <-> file transition for n during these renames.\n[...]\n> \tUnlink the corresponding loose refs so packed-refs\n> \t\tbecomes authoritative for them.\n> \tLock packed-refs.\n> \tPerform updates and removals in the packed-refs cache.\n> \tCommit packed-refs.\n\n... or is the problem that the reflogs conflict?\n\nHow does rename_ref handle propagating the reflog from the old\nname to the new name, by the way?\n\nThanks,\nJonathan\n"},{"id":"242571","messageId":"CAL=YDWnRkwX3dCv9dyZmBdh2ZV7aSUCS=F6w67gJuBmCzGGBwQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140522235351.GB12314@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-23T14:59:21Z","receivedAt":"2014-05-23T14:59:21Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 4:53 PM, Jonathan Nieder wrote:\n> Jonathan Nieder wrote:\n>> Ronnie Sahlberg wrote:\n>\n>>> I hate rename_ref :-)\n>>>\n>>> I have reworked the transaction code to special case the deletion of\n>>> the old ref for n/n -> n and n -> n/n renames\n>>> so that we can carefully avoid n/n.lock files to exist or prevent the\n>>> directory <-> file transition for n during these renames.\n> [...]\n>> Unlink the corresponding loose refs so packed-refs\n>> becomes authoritative for them.\n>> Lock packed-refs.\n>> Perform updates and removals in the packed-refs cache.\n>> Commit packed-refs.\n>\n> ... or is the problem that the reflogs conflict?\n>\n> How does rename_ref handle propagating the reflog from the old\n> name to the new name, by the way?\n\nI haven't touched that yet, but we can fix it after the next series\nwhen we have transaction support for reflogs.\n\nIt still renames the reflog via the magic name\n#define TMP_RENAMED_LOG \"logs/refs/.tmp-renamed-log\"\n\nand thus, if you run two renames at the same time there is a race that\nmight make you end up with the wrong reflog after the rename.\n\n\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"242572","messageId":"537F67DD.5010101@alum.mit.edu","threadId":"36669","inReplyTo":"1400174999-26786-42-git-send-email-sahlberg@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-05-23T15:23:09Z","receivedAt":"2014-05-23T15:23:09Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 05/15/2014 07:29 PM, Ronnie Sahlberg wrote:\n> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.\n> This flag indicates that the ref does not exist as a loose ref andf only as\n> a packed ref. If this is the case we then change the commit code so that\n> we skip taking out a lock file and we skip calling delete_ref_loose.\n> Check for this flag and die(BUG:...) if used with _update or _create.\n> \n> At the start of the transaction, before we even start locking any refs,\n> we add all such REF_ISPACKONLY refs to delnames so that we have a list of\n> all pack only refs that we will be deleting during this transaction.\n\nLet me make a few comments about how I think ref packing fits into the\nscheme for pluggable reference back ends. The comments may or may not\nbe relevant to this particular commit.\n\nWe want to have a reference API that can be implemented by various back\nends. Obviously we need operations to read a reference, enumerate\nexisting references, update references within a transaction, and\nprobably do similar things with symbolic references.\n\nThe status quo is that we have a single reference back end consisting of\nloose references sitting on top of packed references.\n\nBut really, loose references and packed references are two relatively\nindependent reference back ends [1]. We just happen to use them layered\non top of each other.\n\nThis suggests to me that our current structure is best modeled as two\nindependent reference back ends, with a third implementation of the same\nreference API whose job it is to compose the first two. In pseudocode,\n\n interface ReferenceBackend:\n read_ref(refname)\n __iter__(...)\n begin_transaction(...)\n update_reference(...)\n ...\n commit_transaction(...)\n\n class LooseReferenceBackend(ReferenceBackend):\n ...\n\n class PackedReferenceBackend(ReferenceBackend):\n ...\n\n class StackedReferenceBackend(ReferenceBackend):\n def __init__(self, backend1, backend2):\n self.backend1 = backend1\n self.backend2 = backend2\n\n def read_ref(refname):\n try:\n return backend1.read_ref(refname)\n except ReferenceNotFound:\n return backend2.read_ref(refname)\n\n def __iter__(...):\n i1 = self.backend1.iterate()\n i2 = self.backend2.iterate()\n while i1.has_next() and i2.has_next():\n if i1.peek_next() < i2.peek_next():\n yield i1.next()\n elif i1.peek_next() == i2.peek_next():\n yield i1.next()\n i2.next() # discard\n else\n yield i2.next()\n while i1.has_next():\n yield i1.next()\n while i2.has_next():\n yield i2.next()\n\n def pack_refs(...):\n ...\n\net cetera.\n\n>From this point of view it is clear that packing refs is not an\noperation that belongs in the ReferenceBackend API, but rather in the\nStackedReferenceBackend interface. The reason is that it doesn't affect\nhow references appear to the outside world, but rather just reorganizes\nthem between the two backends to which StackedReferenceBackend\ndelegates. The *implementation* of pack_refs() involves adding\nreferences to backend2 and deleting them from backend1. But since\nadding and deleting references *are* part of the ReferenceBackend API,\nStackedReferenceBackend can compose *any arbitrary two* reference\nbackends (including other StackedReferenceBackends). Pruning references\nthat have been packed is also not part of the ReferenceBackend API.\nRather it is a plain old deletion of a reference from backend1 of a\nStackedReferenceBackend.\n\nI haven't thought through all of the ramifications of this design, but\nI'm pretty sure it's where we want to be headed.\n\nMichael\n\n[1] Forget for the sake of this discussion that we can't store symbolic\nreferences as packed refs.\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"242573","messageId":"20140523155353.GA4065@google.com","threadId":"36669","inReplyTo":"537F67DD.5010101@alum.mit.edu","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-23T15:53:53Z","receivedAt":"2014-05-23T15:53:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nMichael Haggerty wrote:\n\n> The status quo is that we have a single reference back end consisting of\n> loose references sitting on top of packed references.\n>\n> But really, loose references and packed references are two relatively\n> independent reference back ends [1]. We just happen to use them layered\n> on top of each other.\n>\n> This suggests to me that our current structure is best modeled as two\n> independent reference back ends, with a third implementation of the same\n> reference API whose job it is to compose the first two.\n[...]\n> [1] Forget for the sake of this discussion that we can't store symbolic\n> references as packed refs.\n\nI find it hard to forget that. ;-) More to the point, the trouble\nwith loose refs and packed refs as independent reference backends is\nthat neither has very good performance characteristics. Enumerating\nmany loose refs is slow. Adding a new packed ref to a large list is\nalso slow. Git currently uses both loose and packed refs in a way\nthat allows each to overcome the limitations of the other, and the\nfact that it involves two on-disk data structures seems to me like an\nimplementation detail of how it achieves that.\n\nSo I believe most git code should not have to know about the\ndifference between loose and packed refs (or the upper and lower\nlayer) to allow the details of the layering can be tuned in low-level\nref handling code.\n\nOn the other hand, from a code structure perspective I can easily\nbelieve that implementing some subset (or maybe even all) of the\nreference backend API for loose refs and packed refs separately and\nproviding a separate file describing how to compose them might be the\ncleanest way to write this code. It's more general layering that\nseems to lie in the direction of madness.\n\nMaybe I'm wrong and people will find lots of use for combinations like\n * loose refs shadowing an sqlite database\n * tdb shadowing mysql\n * etc\nIt's easy to prove a naysayer wrong with code and I don't want to\ndiscourage that.\n\nFor the topic at hand it's relevant because packed-refs have\nproperties that make some operations (certain deletion/ref creation\ncombinations) much less fussy than loose refs, and it would be nice to\nbe able to take advantage of that. In the long term I would like to\nsee git taking advantage of that when someone tries to fetch refs with\nnames that would conflict on the filesystem (e.g., topic, topic/a,\ntopic/b).\n\nThanks,\nJonathan\n"},{"id":"242588","messageId":"20140523182429.GD12314@google.com","threadId":"36669","inReplyTo":"CAL=YDWnRkwX3dCv9dyZmBdh2ZV7aSUCS=F6w67gJuBmCzGGBwQ@mail.gmail.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-05-23T18:24:29Z","receivedAt":"2014-05-23T18:24:29Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ronnie Sahlberg wrote:\n> On Thu, May 22, 2014 at 4:53 PM, Jonathan Nieder wrote:\n\n>> ... or is the problem that the reflogs conflict?\n>>\n>> How does rename_ref handle propagating the reflog from the old\n>> name to the new name, by the way?\n>\n> I haven't touched that yet, but we can fix it after the next series\n> when we have transaction support for reflogs.\n>\n> It still renames the reflog via the magic name\n> #define TMP_RENAMED_LOG \"logs/refs/.tmp-renamed-log\"\n\nOkay, after looking at what \"git branch -m\" currently does (deletes\nthe old ref and creates the new one in separate transactions), I'm\nconvinced that this take-locks-for-a-shorter-period-than-is-necessary\ntrick is an improvement relative to the status quo. It gets rid of\nthe window with objects unreferenced between the ref deletion and ref\ncreation, which is a nice improvement.\n\nI still haven't looked closely at the details of the code change but\nthe idea looks sane as a first step.\n\nLater we can try to get the semantics right for this kind of\ndelete/create pair in general transactions, if someone is interested.\n:) No need for the perfect to be the enemy of the good in the\nmeantime.\n\nThanks,\nJonathan\n"},{"id":"242617","messageId":"537FC180.9020503@alum.mit.edu","threadId":"36669","inReplyTo":"20140523155353.GA4065@google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-05-23T21:45:36Z","receivedAt":"2014-05-23T21:45:36Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 05/23/2014 05:53 PM, Jonathan Nieder wrote:\n> Hi,\n> \n> Michael Haggerty wrote:\n> \n>> The status quo is that we have a single reference back end consisting of\n>> loose references sitting on top of packed references.\n>>\n>> But really, loose references and packed references are two relatively\n>> independent reference back ends [1]. We just happen to use them layered\n>> on top of each other.\n>>\n>> This suggests to me that our current structure is best modeled as two\n>> independent reference back ends, with a third implementation of the same\n>> reference API whose job it is to compose the first two.\n> [...]\n>> [1] Forget for the sake of this discussion that we can't store symbolic\n>> references as packed refs.\n> \n> I find it hard to forget that. ;-) More to the point, the trouble\n> with loose refs and packed refs as independent reference backends is\n> that neither has very good performance characteristics. Enumerating\n> many loose refs is slow. Adding a new packed ref to a large list is\n> also slow. Git currently uses both loose and packed refs in a way\n> that allows each to overcome the limitations of the other, and the\n> fact that it involves two on-disk data structures seems to me like an\n> implementation detail of how it achieves that.\n\nI'm not advocating that we use loose refs or packed refs alone. But I\nlike the code decoupling that this implementation would (I predict) yield.\n\nMy main point was that pack-refs is not an integral part of the\nreference API but rather a tuning feature very specific to the\nloose/packed reference storage scheme.\n\n> So I believe most git code should not have to know about the\n> difference between loose and packed refs (or the upper and lower\n> layer) to allow the details of the layering can be tuned in low-level\n> ref handling code.\n> \n> On the other hand, from a code structure perspective I can easily\n> believe that implementing some subset (or maybe even all) of the\n> reference backend API for loose refs and packed refs separately and\n> providing a separate file describing how to compose them might be the\n> cleanest way to write this code. It's more general layering that\n> seems to lie in the direction of madness.\n> \n> Maybe I'm wrong and people will find lots of use for combinations like\n> * loose refs shadowing an sqlite database\n> * tdb shadowing mysql\n> * etc\n> It's easy to prove a naysayer wrong with code and I don't want to\n> discourage that.\n\nI admit that I don't have any other layered storage schemes in mind.\n\n> For the topic at hand it's relevant because packed-refs have\n> properties that make some operations (certain deletion/ref creation\n> combinations) much less fussy than loose refs, and it would be nice to\n> be able to take advantage of that. In the long term I would like to\n> see git taking advantage of that when someone tries to fetch refs with\n> names that would conflict on the filesystem (e.g., topic, topic/a,\n> topic/b).\n\nA transition to allowing D/F-conflicting reference names has two very\ndistinct aspects to it:\n\n1. Changing to how references (and reflogs!) are stored to make it\ntechnically possible to store such references.\n\n2. Removing restrictions on actually creating such references.\n\nWe can take step 1 any time because it is a purely local decision.\nThough I think you would need a repository format bump to allow it.\nEven though you could work around the D/F problem for references by\npacking problematic ones, it is a kludge with a potentially significant\nperformance cost. And we have the same problem with reflogs, with no\nanalogous kludge. I'd rather make a clean break, for example mapping\nreference names into filenames using some kind of escaping of special\ncharacters and suffixes like \".d\" and \".f\" to prevent directories and\nfiles from conflicting. Maybe (depending on the OS and/or filesystem?)\nescape all non-ASCII characters or even all non-lower-case ASCII\ncharacters to prevent problems with case sensitivity, internal vs.\nfilesystem character encodings, and NFC vs NFD.\n\nBut we won't want to take step 2 until Git clients that are 1-capable\nare widespread; otherwise people with different client versions will\nhave trouble collaborating. Maybe step 2 should be governed by a\nconfiguration option with three settings:\n\n FORBIDDEN - don't allow references with D/F conflicts to exist\n in this repository\n\n NO_CREATE - don't allow the creation of such references locally,\n but accept them from remote sources via commands like \"fetch\".\n This setting could be used to avoid creating problems for\n collaborators.\n\n ALLOWED - no restriction on the creation of references with D/F\n conflicts.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"242759","messageId":"xmqqzji3f55i.fsf@gitster.dls.corp.google.com","threadId":"36669","inReplyTo":"537F67DD.5010101@alum.mit.edu","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-27T18:27:53Z","receivedAt":"2014-05-27T18:27:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> This suggests to me that our current structure is best modeled as two\n> independent reference back ends, with a third implementation of the same\n> reference API whose job it is to compose the first two. In pseudocode,\n> ...\n\nThat is an interesting view.\n\nHow does reflog fit into the picture? Is it a completely\nindependent thing that is called from any implementation of\nReferenceBackend interface?\n\n> From this point of view it is clear that packing refs is not an\n> operation that belongs in the ReferenceBackend API, but rather in the\n> StackedReferenceBackend interface.\n\nWhen an implementation of ReferenceBackend has skewed performance\ncharacteristics (e.g. PackedReferenceBackend really prefers to be\nmodified in bulk), how would that interact with the abstraction?\n\nFor example, when the application does:\n\n begin_transaction()\n for ref in many_refs():\n\tdelete_reference(ref)\n commit_transaction()\n\nStackedReferenceBackend() that consists of Loose on top of Packed\nmay want to implement the commit phase like so:\n\n - tell Packed backend to repack without the deleted refs\n - tell Loose backend to delete the deleted refs\n\nBut the above would not quite work, as somebody needs to remove logs\nfor refs that were only in the Packed backend, and \"repack without\nthese refs\" API supported by the Packed backend cannot be that\nsomebody---\"repack packed-refs without A B C\" cannot unconditionally\nremove logs for A B C without checking if A B C exists as Loose.\n\n\n\n \n"},{"id":"242761","messageId":"CAL=YDWn2HrTUnjKOCa4h6S0M9fxz6hfEgrLGCkvdaUvAD2fAcQ@mail.gmail.com","threadId":"36669","inReplyTo":"20140522192717.GU12314@google.com","subject":"Re: [PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-27T18:37:43Z","receivedAt":"2014-05-27T18:37:43Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 12:27 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -798,11 +798,19 @@ struct name_conflict_cb {\n>> const char *refname;\n>> const char *oldrefname;\n>> const char *conflicting_refname;\n>> + const char **skip;\n>> + int skipnum;\n>\n> Would a struct string_list make sense here? (See\n> Documentation/technical/api-string-list.txt.)\n\nIt would. But it is better to do that change later.\nCurrently we have both repack_without_ref and repack_without_refs that\ntake the same char ** argument.\nAfter next series we will have removed one of these functions and have\nan easier API to modify (once we start tracking the skiplist as part\nof the transaction instead).\n\nInstead of doing this change and then redoing it once we move a lot of\nthe actual work from _commit to _update\nI will do this change towards the end of the next series.\n\n>\n> [...]\n>> };\n>>\n>> static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n>> {\n>> struct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;\n>> + int i;\n>> + for(i = 0; i < data->skipnum; i++) {\n>\n\nFixed.\n\n> (style nit) missing space after 'for'.\n>\n>> + if (!strcmp(entry->name, data->skip[i])) {\n>> + return 0;\n>> + }\n>\n> Style: git tends to avoid braces around a single-line if/for/etc body.\n>\n\nFixed.\n\n> [...]\n>> @@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)\n>> * conflicting with the name of an existing reference in dir. If\n>> * oldrefname is non-NULL, ignore potential conflicts with oldrefname\n>> * (e.g., because oldrefname is scheduled for deletion in the same\n>> - * operation).\n>> + * operation). skip contains a list of refs we want to skip checking for\n>> + * conflicts with. Refs may be skipped due to us knowing that it will\n>> + * be deleted later during a transaction that deletes one reference and then\n>> + * creates a new conflicting reference. For example a rename from m to m/m.\n>\n> This example of \"Refs may be skipped due to\" seems overly complicated.\n> Isn't the idea just that skip contains a list of refs scheduled for\n> deletion in this transaction, since they shouldn't be treated as\n> conflicts at all (for example when renamining m to m/m)?\n>\n> I wonder if there's some way to make use of the result of the naive\n> refname_available check to decide what to do when creating a ref.\n>\n> E.g.: if a refname would be available except there's a ref being\n> deleted in the way, we could do one of the following:\n>\n> a. delete all relevant loose refs and perform the transaction in\n> packed-refs, or\n>\n> b. order operations to avoid the D/F conflict, even with loose refs\n> (the hardest case is if the ref being deleted uses a directory\n> and we want to create a file with the same name. But that's\n> still doable if we're willing to rmdir when needed as part of\n> the loop to commit changes)\n>\n> The packed-refs trick (a) seems much simpler, but either should work.\n>\n> This could be done e.g. by checking is_refname_available with an empty\n> list first before doing the real thing with a list of exclusions.\n>\n> [...]\n>> @@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms\n>> int log = !lstat(git_path(\"logs/%s\", oldrefname), &loginfo);\n>> const char *symref = NULL;\n>>\n>> + if (!strcmp(oldrefname, newrefname))\n>> + return 0;\n>\n> What is the intended result if I try to rename a nonexistent ref or an\n> existent symref to its own name?\n\nYeah, I should get rid of that.\n\nI have renoved the rename_ref patch and moved it to the next series.\nI think we can solve this easier/better once we have the other patches in first.\n\n>\n> Sorry to be so fussy about this part. It's not that I think that this\n> change is trying to do something bad --- in fact, it's more the\n> opposite, that I'm excited to see git learning to have a better\n> understanding and handling of refname D/F conflicts.\n>\n> That would allow:\n>\n> * \"git fetch --prune\" working as a single transaction even if\n> the repository being fetched from removed a refs/heads/topic\n> branch and created refs/heads/topic/1 and refs/heads/topic/2\n>\n> * \"git fast-import\" and \"git fetch --mirror\" learning the same trick\n>\n> * fewer code paths having to be touched to be able to (optionally)\n> let git actually tolerate D/F conflicts, for people who want to\n> have 'topic', 'topic/1', and 'topic/2' branches at the same time.\n> This could be turned on by default for remote-tracking refs. It\n> would be especially nice for people on Windows and Mac OS where\n> there can be D/F conflicts that people on Linux didn't notice due\n> to case-sensitivity.\n>\n> Longer term, through a configuration that starts turned off by\n> default and has the default flipped as more people have upgraded\n> git, this could make D/F conflicts in refnames stop being an error\n> altogether.\n>\n> So it's kind of exciting to see, even though it's fussy to get it\n> right.\n>\n> Thanks,\n> Jonathan\n"},{"id":"242765","messageId":"CAL=YDW=tJaUwRW0gaP-HCEySet8LV0jAu90xy7VXCQK48xz5Nw@mail.gmail.com","threadId":"36669","inReplyTo":"20140522230802.GZ12314@google.com","subject":"Re: [PATCH v8 00/44] Use ref transactions for all ref updates","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-05-27T19:05:21Z","receivedAt":"2014-05-27T19:05:21Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, May 22, 2014 at 4:08 PM, Jonathan Nieder wrote:\n> Ronnie Sahlberg wrote:\n>\n>> This patch series can also be found at\n>> https://github.com/rsahlberg/git/tree/ref-transactions\n>\n> Continuing with the review of 65a1cb7b (2014-05-22 12:08):\n>\n> 11/40 change ref_transaction_update() to do error checking and return status\n> The \"there will be in the future\" sounds ominous. Do you have an\n> example in mind? E.g., I suppose it would be nice if _update could\n> notice D/F conflicts or connection to a database server closing early,\n> but it's not clear to me whether the kind of errors you're talking\n> about are that or something else.\n>\n\nUpdated the message.\nNext series moves both locking as well as checking for name conflicts\nto _update.\n\n> With or without such a clarification,\n> Reviewed-by: Jonathan Nieder \n>\n> 12/40 change ref_transaction_create to do error checking and return status\n> What does \"On failure the err buffer will be updated\" mean? Will\n> it clear err and replace it with a message, append to err, or\n> something else? Does the message explain the context or is the\n> caller responsible for adding to it? Does the message end with a\n> newline or is the caller responsible for adding one when printing it\n> out?\n\nI have updated the documentation.\nMessage is appended to the string buffer. Caller is required to\nstrbuf_reset before calling the transaction if caller wants only most\nrecent error instead of all errors appended one by one.\n\n\n>\n> For cases like this where lots of functions have a similar API,\n> API comments start to become potentially repetitive. It might be\n> better to explain conventions at the top of the file or in\n> Documentation/technical/api-refs.txt and say \"See the top of the\n> file for error handling conventions\" or \"Returns non-zero and\n> appends a message to err on error. See\n> Documentation/technical/api-refs.txt for more details on error\n> handling\".\n\nDone.\n\n>\n> 13/40 ref_transaction_delete to check for error and return status\n> Each successive commit dropped something from its subject. :)\n> (First the (), then the verb.)\n\nDone.\n\n>\n> Same comments as before about an example being useful for the\n> log message and the API documentation on error handling being a\n> bit vague.\n>\n> 14/40 make ref_transaction_begin take an err argument\n> I found the \"failed to connect to mysql\" example instructive while\n> doing reviews. Perhaps it would be worth mentioning in the commit\n> message.\n>\n> Reviewed-by: Jonathan Nieder \n>\n> 15/40 add transaction.status and track OPEN/CLOSED/ERROR\n> It says an ERRORed transaction cannot be committed and can be rolled\n> back by calling _free. Can a CLOSED transaction be committed or\n> _freed?\n>\n> What does \"faild\" mean in the documentation comments? (Maybe\n> \"non-OPEN\"?)\n>\n> In the previous version of this patch passing a non-OPEN transaction\n> would die(\"BUG: \"...) to diagnose the caller's mistake. Now I'm\n> confused about the API: it seems you're allowed to pass a non-OPEN\n> transaction but it doesn't append a message to 'err' in that case.\n> Is this meant as a way to save the caller some typing, like\n> fwrite/fclose do? (I've found people often make mistakes with the\n> fwrite API fwiw but can understand the appeal of it.)\n>\n> Maybe with more context I'd like this. As is, it feels like a step\n> in the wrong direction.\n>\n> 16/40 tag: use ref transactions when doing updates\n> Reviewed-by: Jonathan Nieder \n>\n> 17/40 replace: use ref transactions when doing updates\n> Reviewed-by: Jonathan Nieder \n>\n> 18/40 commit: use ref transactions for updates\n> Reviewed-by: Jonathan Nieder \n>\n> 19/40 sequencer: use ref transactions for all ref updates\n> This would be a lot simpler if the \"ref_transaction_commit should not\n> free the transaction\" patch came before it (yes, sorry, killing the\n> fun). I can push the result of a rebase doing that somewhere if you\n> like.\n\nBeeing done.\n\n>\n> 20/40 fast-import: change update_branch to use ref transactions\n> Likewise.\n"},{"id":"242858","messageId":"5385F0E5.4010306@alum.mit.edu","threadId":"36669","inReplyTo":"xmqqzji3f55i.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-05-28T14:21:25Z","receivedAt":"2014-05-28T14:21:25Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 05/27/2014 08:27 PM, Junio C Hamano wrote:\n> Michael Haggerty writes:\n> \n>> This suggests to me that our current structure is best modeled as two\n>> independent reference back ends, with a third implementation of the same\n>> reference API whose job it is to compose the first two. In pseudocode,\n>> ...\n> \n> That is an interesting view.\n> \n> How does reflog fit into the picture? Is it a completely\n> independent thing that is called from any implementation of\n> ReferenceBackend interface?\n\nThat's a good question. It certainly wouldn't work for each of the\nloose and packed backends to try to do logging. I think the best\nsolution would be to have a logging wrapper as explained below.\n\n>> From this point of view it is clear that packing refs is not an\n>> operation that belongs in the ReferenceBackend API, but rather in the\n>> StackedReferenceBackend interface.\n> \n> When an implementation of ReferenceBackend has skewed performance\n> characteristics (e.g. PackedReferenceBackend really prefers to be\n> modified in bulk), how would that interact with the abstraction?\n> \n> For example, when the application does:\n> \n> begin_transaction()\n> for ref in many_refs():\n> \tdelete_reference(ref)\n> commit_transaction()\n> \n> StackedReferenceBackend() that consists of Loose on top of Packed\n> may want to implement the commit phase like so:\n> \n> - tell Packed backend to repack without the deleted refs\n> - tell Loose backend to delete the deleted refs\n\nI think for any two backends that are stacked, you would need to break\ndown a transaction as follows (here generalized to include not only\ndeletions):\n\n packed->begin_transaction()\n loose->begin_transaction()\n\n # And this part is done within stacked->commit_transaction():\n for entry in many_ref_updates():\n if have_old:\n stacked->verify_reference(ref, old_sha1)\n\n if entry is a delete:\n packed->delete_reference(entry)\n loose->update_reference(entry)\n\n if (!packed->commit_transaction())\n loose->commit_transaction()\n\nVerifying old values is impossible to do batchwise with the current API,\nbecause the old value of the packed ref has to be verified if and only\nif there is no corresponding loose ref.\n\n> But the above would not quite work, as somebody needs to remove logs\n> for refs that were only in the Packed backend, and \"repack without\n> these refs\" API supported by the Packed backend cannot be that\n> somebody---\"repack packed-refs without A B C\" cannot unconditionally\n> remove logs for A B C without checking if A B C exists as Loose.\n\nCorrect. That's another reason that logging has to be the\nresponsibility of something at the \"stacked\" level of abstraction or higher.\n\nI think the logging should be done by yet another outer layer of wrapper\nthat only does the logging, while also passing all updates down 1:1 to\nthe backend that it wraps (which in our case would be a stacked\nbackend). Then the loose and packed backends could remain completely\nignorant of the fact that reference updates can be logged. I think the\nlogging layer could implement the same reference backend API as the\nother backends.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"242869","messageId":"xmqqk395et7l.fsf@gitster.dls.corp.google.com","threadId":"36669","inReplyTo":"5385F0E5.4010306@alum.mit.edu","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-28T16:58:06Z","receivedAt":"2014-05-28T16:58:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> I think for any two backends that are stacked, you would need to break\n> down a transaction as follows (here generalized to include not only\n> deletions):\n>\n> packed->begin_transaction()\n> loose->begin_transaction()\n>\n> # And this part is done within stacked->commit_transaction():\n> for entry in many_ref_updates():\n> if have_old:\n> stacked->verify_reference(ref, old_sha1)\n>\n> if entry is a delete:\n> packed->delete_reference(entry)\n> loose->update_reference(entry)\n>\n> if (!packed->commit_transaction())\n> loose->commit_transaction()\n\nUgggly. In general you would need to worry about the case where the\nfirst commit succeeds and then the second commit fails, wouldn't\nyou?\n\nThe above happens not to break horribly wrong for the \"Loose on top\nof Packed\" layering, in that the refs you wanted to delete are only\ngone from Packed but still are in Loose, and the end result would be\nsome of them are really gone (because they weren't in Loose) and\nsome others remain (because they were in Loose), and at least you\ndid not lose any ref you did not want to discard. But it still is\nnot what should happen in a proper \"transaction\".\n\n>> But the above would not quite work, as somebody needs to remove logs\n>> for refs that were only in the Packed backend, and \"repack without\n>> these refs\" API supported by the Packed backend cannot be that\n>> somebody---\"repack packed-refs without A B C\" cannot unconditionally\n>> remove logs for A B C without checking if A B C exists as Loose.\n>\n> Correct. That's another reason that logging has to be the\n> responsibility of something at the \"stacked\" level of abstraction or higher.\n>\n> I think the logging should be done by yet another outer layer of\n> wrapper that only does the logging, while also passing all updates\n> down 1:1 to the backend that it wraps (which in our case would be\n> a stacked backend). ... Then the loose and packed backends could\n> remain completely ignorant of the fact that reference updates can\n> be logged.\n\nThat would mean that Loose (or Packed) class cannot be used as-is\nand always needs to be wrapped with the layer that does the logging,\nno? In a system with \"only packed-refs, no loose\", you would want\nPacked.deleteRefs() to remove the named refs ref and their reflogs,\nbut that would mean that the Layered wrapper that uses Loose\noverlayed on Packed cannot call that method, because it does not\nwant reflogs of the refs in Packed covered by the ones in Loose.\n"},{"id":"242914","messageId":"538661F9.50205@alum.mit.edu","threadId":"36669","inReplyTo":"xmqqk395et7l.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-05-28T22:23:53Z","receivedAt":"2014-05-28T22:23:53Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 05/28/2014 06:58 PM, Junio C Hamano wrote:\n> Michael Haggerty writes:\n> \n>> I think for any two backends that are stacked, you would need to break\n>> down a transaction as follows (here generalized to include not only\n>> deletions):\n>>\n>> packed->begin_transaction()\n>> loose->begin_transaction()\n>>\n>> # And this part is done within stacked->commit_transaction():\n>> for entry in many_ref_updates():\n>> if have_old:\n>> stacked->verify_reference(ref, old_sha1)\n>>\n>> if entry is a delete:\n>> packed->delete_reference(entry)\n>> loose->update_reference(entry)\n>>\n>> if (!packed->commit_transaction())\n>> loose->commit_transaction()\n> \n> Ugggly.\n\nThe ugliness would be encapsulated in the stacked reference backend. It\nwouldn't have to appear in client code.\n\n> In general you would need to worry about the case where the\n> first commit succeeds and then the second commit fails, wouldn't\n> you?\n> \n> The above happens not to break horribly wrong for the \"Loose on top\n> of Packed\" layering, in that the refs you wanted to delete are only\n> gone from Packed but still are in Loose, and the end result would be\n> some of them are really gone (because they weren't in Loose) and\n> some others remain (because they were in Loose), and at least you\n> did not lose any ref you did not want to discard. But it still is\n> not what should happen in a proper \"transaction\".\n\nTrue, but we don't have proper transactions anyway. If anything goes\nwrong when renaming one of the loose reference lockfiles, we have no way\nof reliably rolling back the loose references that have already been\nchanged. The word \"transaction\" as we use it really only suggests that\nwe are doing our best to change the references all-or-nothing even in\nthe face of concurrent modifications by other processes. It certainly\ndoesn't protect against power failures and stuff like that.\n\nI suppose if you wanted to make reference deletions a little bit more\ntransaction-like, you could split them into two steps to \"loosen\" any\npacked references and a deletion step that deletes the loose references.\n The advantage of this scheme is that each step requires a transaction\non only a single back end at a time, though it requires part or all of\nthe other back end to be locked simultaneously.\n\n # The first two iterations \"loosen\" any packed references\n # corresponding to references that are to be deleted. It is purely\n # an internal reorganization that doesn't change the externally-\n # visible values of any references and can be done within a\n # separate transaction:\n for each reference to delete:\n if reference exists packed but not loose:\n create a loose ref with the value from the packed ref\n for each reference to delete:\n if reference exists packed:\n delete packed version of reference\n\n # Now we only have to delete the loose version of the references.\n # This should be done after activating the packed reference file\n # but while continuing to hold the packed-refs lock:\n for each reference to delete:\n delete loose version of reference\n\nI'd have to think more about whether this all really works generically\nwithout requiring lots of extra round-trips to a possibly-remote\nbackend. But I'm not sure we really need this composability in its full\ngenerality. For example, maybe it is impossible to stack a low-latency\nand a high-latency backend together while minimizing round-trips to the\nlatter.\n\n>>> But the above would not quite work, as somebody needs to remove logs\n>>> for refs that were only in the Packed backend, and \"repack without\n>>> these refs\" API supported by the Packed backend cannot be that\n>>> somebody---\"repack packed-refs without A B C\" cannot unconditionally\n>>> remove logs for A B C without checking if A B C exists as Loose.\n>>\n>> Correct. That's another reason that logging has to be the\n>> responsibility of something at the \"stacked\" level of abstraction or higher.\n>>\n>> I think the logging should be done by yet another outer layer of\n>> wrapper that only does the logging, while also passing all updates\n>> down 1:1 to the backend that it wraps (which in our case would be\n>> a stacked backend). ... Then the loose and packed backends could\n>> remain completely ignorant of the fact that reference updates can\n>> be logged.\n> \n> That would mean that Loose (or Packed) class cannot be used as-is\n> and always needs to be wrapped with the layer that does the logging,\n> no?\n\nCorrect.\n\n> In a system with \"only packed-refs, no loose\", you would want\n> Packed.deleteRefs() to remove the named refs ref and their reflogs,\n> but that would mean that the Layered wrapper that uses Loose\n> overlayed on Packed cannot call that method, because it does not\n> want reflogs of the refs in Packed covered by the ones in Loose.\n\nThe two scenarios would be roughly\n\n refhandler = new LoggingWrapper(new PackedRefs())\n\nand\n\n refhandler = new LoggingWrapper(\n new StackedRefs(new LooseRefs(), new PackedRefs())\n )\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"}]}