{"thread":{"id":"41344","subject":"[PATCH v4 01/21] refs: add a backend method structure with transaction functions","startedAt":"2016-02-05T19:44:01Z","lastAt":"2016-02-18T18:32:37Z","messageCount":53,"participants":["David Turner","Junio C Hamano","Michael Haggerty","Duy Nguyen"],"isPatch":true,"patchVersion":4,"patchTotal":21},"messages":[{"id":"277568","messageId":"1454701462-3817-1-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":null,"subject":"[PATCH v4 00/20] refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:01Z","receivedAt":"2016-02-05T19:44:01Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Changes to this version:\nre-rolled on top of pu as-of 9db66d9f1aa.\n\nBug fixes include:\nFor submodules: memory leaks; segfault on bad config. (thanks to Peff)\nIn symref splitting: check that would always succeed (thanks to Peff)\nA bogus double-declaration of a var (thanks to Ramsay Jones)\nTwo memory leaks (thanks to Thomas Gummerer)\nAn unused var (thanks to Duy Nguyen)\n\nOther improvements are:\nStrings prepped for 18n (thanks to Duy Nguyen)\nCleaner submodule handling (thanks to Peff)\nWhitelisting instead of blacklisting in git-new-workdir (thanks to\n Thomas Gummerer)\nAllow older gits to recognize lmdb-backend git repos (thanks to Duy Nguyen)\nTab completion and cleaner commit messages (thanks SZEDER Gábor)\nRemoved some #ifdefs, moving all backend setup to one place (thanks to Duy\nNguyen)\n\nThanks to all for reviews.\n\nDavid Turner (18):\n refs: add do_for_each_per_worktree_ref\n refs: add methods for reflog\n refs: add method for initial ref transaction commit\n refs: add method for delete_refs\n refs: add methods to init refs db\n refs: add method to rename refs\n refs: make lock generic\n refs: move duplicate check to common code\n refs: allow log-only updates\n refs: resolve symbolic refs first\n refs: always handle non-normal refs in files backend\n init: allow alternate ref strorage to be set for new repos\n refs: check submodules ref storage config\n clone: allow ref storage backend to be set for clone\n svn: learn ref-storage argument\n refs: add register_ref_storage_backends()\n refs: add LMDB refs storage backend\n refs: tests for lmdb backend\n\nRonnie Sahlberg (3):\n refs: add a backend method structure with transaction functions\n refs: add methods for misc ref operations\n refs: add methods for the ref iterators\n\n .gitignore | 1 +\n Documentation/config.txt | 7 +\n Documentation/git-clone.txt | 6 +\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 7 +-\n Documentation/technical/refs-lmdb-backend.txt | 52 +\n Documentation/technical/repository-version.txt | 5 +\n Makefile | 12 +\n builtin/clone.c | 5 +\n builtin/init-db.c | 57 +-\n builtin/submodule--helper.c | 2 +-\n cache.h | 2 +\n config.c | 25 +\n configure.ac | 33 +\n contrib/completion/git-completion.bash | 6 +-\n contrib/workdir/git-new-workdir | 3 +\n git-submodule.sh | 13 +\n git-svn.perl | 6 +-\n path.c | 29 +-\n refs.c | 486 +++++-\n refs.h | 21 +\n refs/files-backend.c | 404 ++---\n refs/lmdb-backend.c | 2052 ++++++++++++++++++++++++\n refs/refs-internal.h | 128 +-\n setup.c | 23 +-\n t/t0001-init.sh | 25 +\n t/t1460-refs-lmdb-backend.sh | 1109 +++++++++++++\n t/t1470-refs-lmdb-backend-reflog.sh | 359 +++++\n t/t1480-refs-lmdb-submodule.sh | 85 +\n t/test-lib.sh | 1 +\n test-refs-lmdb-backend.c | 64 +\n transport.c | 7 +-\n 32 files changed, 4825 insertions(+), 212 deletions(-)\n create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n create mode 100644 refs/lmdb-backend.c\n create mode 100755 t/t1460-refs-lmdb-backend.sh\n create mode 100755 t/t1470-refs-lmdb-backend-reflog.sh\n create mode 100755 t/t1480-refs-lmdb-submodule.sh\n create mode 100644 test-refs-lmdb-backend.c\n\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277566","messageId":"1454701462-3817-2-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 01/21] refs: add a backend method structure with transaction functions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:02Z","receivedAt":"2016-02-05T19:44:02Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd a ref structure for storage backend methods. Start by adding a\nmethod pointer for the transaction commit function.\n\nAdd a function set_refs_backend to switch between storage\nbackends. The files based storage backend is the default.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 40 ++++++++++++++++++++++++++++++++++++++++\n refs.h | 7 +++++++\n refs/files-backend.c | 10 ++++++++--\n refs/refs-internal.h | 12 ++++++++++++\n 4 files changed, 67 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex e2d34b2..1c646f5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -10,6 +10,39 @@\n #include \"tag.h\"\n \n /*\n+ * We always have a files backend and it is the default.\n+ */\n+static struct ref_storage_be *the_refs_backend = &refs_be_files;\n+/*\n+ * List of all available backends\n+ */\n+static struct ref_storage_be *refs_backends = &refs_be_files;\n+\n+int set_ref_storage_backend(const char *name)\n+{\n+\tstruct ref_storage_be *be;\n+\n+\tfor (be = refs_backends; be; be = be->next)\n+\t\tif (!strcmp(be->name, name)) {\n+\t\t\tthe_refs_backend = be;\n+\t\t\treturn 0;\n+\t\t}\n+\treturn 1;\n+}\n+\n+int ref_storage_backend_exists(const char *name)\n+{\n+\tstruct ref_storage_be *be;\n+\n+\tfor (be = refs_backends; be; be = be->next)\n+\t\tif (!strcmp(be->name, name)) {\n+\t\t\tthe_refs_backend = be;\n+\t\t\treturn 1;\n+\t\t}\n+\treturn 0;\n+}\n+\n+/*\n * How to handle various characters in refnames:\n * 0: An acceptable character for refs\n * 1: End-of-component\n@@ -1082,3 +1115,10 @@ int rename_ref_available(const char *oldname, const char *newname)\n \tstrbuf_release(&err);\n \treturn ret;\n }\n+\n+/* backend functions */\n+int ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_commit(transaction, err);\n+}\ndiff --git a/refs.h b/refs.h\nindex 3c3da29..5bc3fbc 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -508,4 +508,11 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n+/*\n+ * Switch to an alternate ref storage backend.\n+ */\n+int set_ref_storage_backend(const char *name);\n+\n+int ref_storage_backend_exists(const char *name);\n+\n #endif /* REFS_H */\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b569762..2a73564 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3146,8 +3146,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3533,3 +3533,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \tunlock_ref(lock);\n \treturn -1;\n }\n+\n+struct ref_storage_be refs_be_files = {\n+\tNULL,\n+\t\"files\",\n+\tfiles_transaction_commit,\n+};\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex c7dded3..fc0e852 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -197,4 +197,16 @@ const char *find_descendant_ref(const char *dirname,\n \n int rename_ref_available(const char *oldname, const char *newname);\n \n+/* refs backends */\n+typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err);\n+\n+struct ref_storage_be {\n+\tstruct ref_storage_be *next;\n+\tconst char *name;\n+\tref_transaction_commit_fn *transaction_commit;\n+};\n+\n+extern struct ref_storage_be refs_be_files;\n+\n #endif /* REFS_REFS_INTERNAL_H */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277567","messageId":"1454701462-3817-3-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 02/21] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:03Z","receivedAt":"2016-02-05T19:44:03Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd ref backend methods for:\nresolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,\ncreate_symref, resolve_gitlink_ref.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n builtin/init-db.c | 1 +\n refs.c | 36 ++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 33 +++++++++++++++++++++++----------\n refs/refs-internal.h | 23 +++++++++++++++++++++++\n 4 files changed, 83 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 07229d6..26e1cc3 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -8,6 +8,7 @@\n #include \"builtin.h\"\n #include \"exec_cmd.h\"\n #include \"parse-options.h\"\n+#include \"refs.h\"\n \n #ifndef DEFAULT_GIT_TEMPLATE_DIR\n #define DEFAULT_GIT_TEMPLATE_DIR \"/usr/share/git-core/templates\"\ndiff --git a/refs.c b/refs.c\nindex 1c646f5..afdde7d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1122,3 +1122,39 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n {\n \treturn the_refs_backend->transaction_commit(transaction, err);\n }\n+\n+const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n+\t\t\t unsigned char *sha1, int *flags)\n+{\n+\treturn the_refs_backend->resolve_ref_unsafe(ref, resolve_flags, sha1,\n+\t\t\t\t\t\t flags);\n+}\n+\n+int verify_refname_available(const char *refname, struct string_list *extra,\n+\t\t\t struct string_list *skip, struct strbuf *err)\n+{\n+\treturn the_refs_backend->verify_refname_available(refname, extra, skip, err);\n+}\n+\n+int pack_refs(unsigned int flags)\n+{\n+\treturn the_refs_backend->pack_refs(flags);\n+}\n+\n+int peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\treturn the_refs_backend->peel_ref(refname, sha1);\n+}\n+\n+int create_symref(const char *ref_target, const char *refs_heads_master,\n+\t\t const char *logmsg)\n+{\n+\treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n+\t\t\t\t\t logmsg);\n+}\n+\n+int resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\tunsigned char *sha1)\n+{\n+\treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 2a73564..882d830 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1343,7 +1343,8 @@ static int resolve_gitlink_ref_recursive(struct ref_cache *refs,\n \treturn resolve_gitlink_ref_recursive(refs, p, sha1, recursion+1);\n }\n \n-int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *sha1)\n+static int files_resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n {\n \tint len = strlen(path), retval;\n \tstruct strbuf submodule = STRBUF_INIT;\n@@ -1583,8 +1584,10 @@ static const char *resolve_ref_1(const char *refname,\n \t}\n }\n \n-const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n-\t\t\t unsigned char *sha1, int *flags)\n+static const char *files_resolve_ref_unsafe(const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n {\n \tstatic struct strbuf sb_refname = STRBUF_INIT;\n \tstruct strbuf sb_contents = STRBUF_INIT;\n@@ -1633,7 +1636,7 @@ static enum peel_status peel_entry(struct ref_entry *entry, int repeel)\n \treturn status;\n }\n \n-int peel_ref(const char *refname, unsigned char *sha1)\n+static int files_peel_ref(const char *refname, unsigned char *sha1)\n {\n \tint flag;\n \tunsigned char base[20];\n@@ -2270,7 +2273,7 @@ static void prune_refs(struct ref_to_prune *r)\n \t}\n }\n \n-int pack_refs(unsigned int flags)\n+static int files_pack_refs(unsigned int flags)\n {\n \tstruct pack_refs_cb_data cbdata;\n \n@@ -2461,10 +2464,10 @@ out:\n \treturn ret;\n }\n \n-int verify_refname_available(const char *newname,\n-\t\t\t struct string_list *extras,\n-\t\t\t struct string_list *skip,\n-\t\t\t struct strbuf *err)\n+static int files_verify_refname_available(const char *newname,\n+\t\t\t\t\t struct string_list *extras,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tstruct ref_dir *packed_refs = get_packed_refs(&ref_cache);\n \tstruct ref_dir *loose_refs = get_loose_refs(&ref_cache);\n@@ -2884,7 +2887,9 @@ static int create_symref_locked(struct ref_lock *lock, const char *refname,\n \treturn 0;\n }\n \n-int create_symref(const char *refname, const char *target, const char *logmsg)\n+static int files_create_symref(const char *refname,\n+\t\t\t const char *target,\n+\t\t\t const char *logmsg)\n {\n \tstruct strbuf err = STRBUF_INIT;\n \tstruct ref_lock *lock;\n@@ -3538,4 +3543,12 @@ struct ref_storage_be refs_be_files = {\n \tNULL,\n \t\"files\",\n \tfiles_transaction_commit,\n+\n+\tfiles_pack_refs,\n+\tfiles_peel_ref,\n+\tfiles_create_symref,\n+\n+\tfiles_resolve_ref_unsafe,\n+\tfiles_verify_refname_available,\n+\tfiles_resolve_gitlink_ref,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fc0e852..e83bc22 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -201,10 +201,33 @@ int rename_ref_available(const char *oldname, const char *newname);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n+/* misc methods */\n+typedef int pack_refs_fn(unsigned int flags);\n+typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n+typedef int create_symref_fn(const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg);\n+\n+/* resolution methods */\n+typedef const char *resolve_ref_unsafe_fn(const char *ref,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags);\n+typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);\n+typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1);\n+\n struct ref_storage_be {\n \tstruct ref_storage_be *next;\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n+\n+\tpack_refs_fn *pack_refs;\n+\tpeel_ref_fn *peel_ref;\n+\tcreate_symref_fn *create_symref;\n+\n+\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n+\tverify_refname_available_fn *verify_refname_available;\n+\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n };\n \n extern struct ref_storage_be refs_be_files;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277570","messageId":"1454701462-3817-4-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 03/21] refs: add methods for the ref iterators","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:04Z","receivedAt":"2016-02-05T19:44:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n 3 files changed, 112 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex afdde7d..e598b73 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1158,3 +1158,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n {\n \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n }\n+\n+int head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref(fn, cb_data);\n+}\n+\n+int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref(fn, cb_data);\n+}\n+\n+int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in(prefix, fn, cb_data);\n+}\n+\n+int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data,\n+\t\t\tunsigned int broken)\n+{\n+\treturn the_refs_backend->for_each_fullref_in(prefix, fn, cb_data,\n+\t\t\t\t\t\t broken);\n+}\n+\n+int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n+\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in_submodule(submodule, prefix,\n+\t\t\t\t\t\t\t fn, cb_data);\n+}\n+\n+int for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_rawref(fn, cb_data);\n+}\n+\n+int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_namespaced_ref(fn, cb_data);\n+}\n+\n+int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 882d830..0772a02 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1774,32 +1774,36 @@ static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n \treturn 0;\n }\n \n-int head_ref(each_ref_fn fn, void *cb_data)\n+static int files_head_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_head_ref(NULL, fn, cb_data);\n }\n \n-int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_head_ref(submodule, fn, cb_data);\n }\n \n-int for_each_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t\tvoid *cb_data)\n {\n \treturn do_for_each_ref(get_ref_cache(submodule), \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, prefix, fn, strlen(prefix), 0, cb_data);\n }\n \n-int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)\n+static int files_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n {\n \tunsigned int flag = 0;\n \n@@ -1808,19 +1812,21 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig\n \treturn do_for_each_ref(&ref_cache, prefix, fn, 0, flag, cb_data);\n }\n \n-int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n-\t\teach_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn,\n+\t\t\t strlen(prefix), 0, cb_data);\n }\n \n-int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, git_replace_ref_base, fn,\n \t\t\t strlen(git_replace_ref_base), 0, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n@@ -1830,7 +1836,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \treturn ret;\n }\n \n-int for_each_rawref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_rawref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0,\n \t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n@@ -3551,4 +3557,15 @@ struct ref_storage_be refs_be_files = {\n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\n \tfiles_resolve_gitlink_ref,\n+\n+\tfiles_head_ref,\n+\tfiles_head_ref_submodule,\n+\tfiles_for_each_ref,\n+\tfiles_for_each_ref_submodule,\n+\tfiles_for_each_ref_in,\n+\tfiles_for_each_fullref_in,\n+\tfiles_for_each_ref_in_submodule,\n+\tfiles_for_each_rawref,\n+\tfiles_for_each_namespaced_ref,\n+\tfiles_for_each_replace_ref,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex e83bc22..4a1d215 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -216,6 +216,24 @@ typedef int verify_refname_available_fn(const char *refname, struct string_list\n typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n \n+/* iteration methods */\n+typedef int head_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int head_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_ref_in_fn(const char *prefix, each_ref_fn fn,\n+\t\t\t void *cb_data);\n+typedef int for_each_fullref_in_fn(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken);\n+typedef int for_each_ref_in_submodule_fn(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data);\n+typedef int for_each_rawref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_namespaced_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n+\n struct ref_storage_be {\n \tstruct ref_storage_be *next;\n \tconst char *name;\n@@ -228,6 +246,17 @@ struct ref_storage_be {\n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n \tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n+\n+\thead_ref_fn *head_ref;\n+\thead_ref_submodule_fn *head_ref_submodule;\n+\tfor_each_ref_fn *for_each_ref;\n+\tfor_each_ref_submodule_fn *for_each_ref_submodule;\n+\tfor_each_ref_in_fn *for_each_ref_in;\n+\tfor_each_fullref_in_fn *for_each_fullref_in;\n+\tfor_each_ref_in_submodule_fn *for_each_ref_in_submodule;\n+\tfor_each_rawref_fn *for_each_rawref;\n+\tfor_each_namespaced_ref_fn *for_each_namespaced_ref;\n+\tfor_each_replace_ref_fn *for_each_replace_ref;\n };\n \n extern struct ref_storage_be refs_be_files;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277585","messageId":"1454701462-3817-5-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 04/21] refs: add do_for_each_per_worktree_ref","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:05Z","receivedAt":"2016-02-05T19:44:05Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might still use files to store per-worktree\nrefs. So the files backend's ref-loading infrastructure should be\navailable to those backends, just for use on per-worktree refs. Add\ndo_for_each_per_worktree_ref, which iterates over per-worktree refs.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 15 ++++++++++++---\n refs/refs-internal.h | 10 ++++++++++\n 2 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0772a02..3d678ad 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -518,9 +518,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n \tdir->sorted = dir->nr = i;\n }\n \n-/* Include broken references in a do_for_each_ref*() iteration: */\n-#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n-\n /*\n * Return true iff the reference described by entry can be resolved to\n * an object in the database. Emit a warning if the referred-to\n@@ -568,6 +565,10 @@ static int do_one_ref(struct ref_entry *entry, void *cb_data)\n \tstruct ref_entry *old_current_ref;\n \tint retval;\n \n+\tif (data->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t ref_type(entry->name) != REF_TYPE_PER_WORKTREE)\n+\t\treturn 0;\n+\n \tif (!starts_with(entry->name, data->base))\n \t\treturn 0;\n \n@@ -1756,6 +1757,14 @@ static int do_for_each_ref(struct ref_cache *refs, const char *base,\n \treturn do_for_each_entry(refs, base, do_one_ref, &data);\n }\n \n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(get_ref_cache(submodule), base, fn, trim,\n+\t\t\t flags | DO_FOR_EACH_PER_WORKTREE_ONLY, cb_data);\n+}\n+\n static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n \tstruct object_id oid;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 4a1d215..cb949c5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -57,6 +57,16 @@\n */\n int refname_is_safe(const char *refname);\n \n+/* Include broken references in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n+\n+/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n+\n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data);\n+\n enum peel_status {\n \t/* object was peeled successfully: */\n \tPEEL_PEELED = 0,\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277571","messageId":"1454701462-3817-6-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 05/21] refs: add methods for reflog","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:06Z","receivedAt":"2016-02-05T19:44:06Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In the file-based backend, the reflog piggybacks on the ref lock.\nSince other backends won't have the same sort of ref lock, ref backends\nmust also handle reflogs.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 36 ++++++++++++++++++++++++------------\n refs/refs-internal.h | 27 +++++++++++++++++++++++++++\n 3 files changed, 97 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex e598b73..3254378 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1212,3 +1212,49 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n }\n+\n+int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,\n+\t\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent_reverse(refname, fn,\n+\t\t\t\t\t\t\t cb_data);\n+}\n+\n+int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,\n+\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent(refname, fn, cb_data);\n+}\n+\n+int for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog(fn, cb_data);\n+}\n+\n+int reflog_exists(const char *refname)\n+{\n+\treturn the_refs_backend->reflog_exists(refname);\n+}\n+\n+int safe_create_reflog(const char *refname, int force_create,\n+\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->create_reflog(refname, force_create, err);\n+}\n+\n+int delete_reflog(const char *refname)\n+{\n+\treturn the_refs_backend->delete_reflog(refname);\n+}\n+\n+int reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\treturn the_refs_backend->reflog_expire(refname, sha1, flags,\n+\t\t\t\t\t prepare_fn, should_prune_fn,\n+\t\t\t\t\t cleanup_fn, policy_cb_data);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 3d678ad..b3372e6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2667,7 +2667,8 @@ static int log_ref_setup(const char *refname, struct strbuf *logfile, struct str\n }\n \n \n-int safe_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+static int files_create_reflog(const char *refname, int force_create,\n+\t\t\t struct strbuf *err)\n {\n \tint ret;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -2923,7 +2924,7 @@ static int files_create_symref(const char *refname,\n \treturn ret;\n }\n \n-int reflog_exists(const char *refname)\n+static int files_reflog_exists(const char *refname)\n {\n \tstruct stat st;\n \n@@ -2931,7 +2932,7 @@ int reflog_exists(const char *refname)\n \t\tS_ISREG(st.st_mode);\n }\n \n-int delete_reflog(const char *refname)\n+static int files_delete_reflog(const char *refname)\n {\n \treturn remove_path(git_path(\"logs/%s\", refname));\n }\n@@ -2975,7 +2976,9 @@ static char *find_beginning_of_line(char *bob, char *scan)\n \treturn scan;\n }\n \n-int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tFILE *logfp;\n@@ -3077,7 +3080,8 @@ int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void\n \treturn ret;\n }\n \n-int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn, void *cb_data)\n {\n \tFILE *logfp;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -3139,7 +3143,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data\n \treturn retval;\n }\n \n-int for_each_reflog(each_ref_fn fn, void *cb_data)\n+static int files_for_each_reflog(each_ref_fn fn, void *cb_data)\n {\n \tint retval;\n \tstruct strbuf name;\n@@ -3449,12 +3453,12 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \treturn 0;\n }\n \n-int reflog_expire(const char *refname, const unsigned char *sha1,\n-\t\t unsigned int flags,\n-\t\t reflog_expiry_prepare_fn prepare_fn,\n-\t\t reflog_expiry_should_prune_fn should_prune_fn,\n-\t\t reflog_expiry_cleanup_fn cleanup_fn,\n-\t\t void *policy_cb_data)\n+static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n {\n \tstatic struct lock_file reflog_lock;\n \tstruct expire_reflog_cb cb;\n@@ -3559,6 +3563,14 @@ struct ref_storage_be refs_be_files = {\n \t\"files\",\n \tfiles_transaction_commit,\n \n+\tfiles_for_each_reflog_ent,\n+\tfiles_for_each_reflog_ent_reverse,\n+\tfiles_for_each_reflog,\n+\tfiles_reflog_exists,\n+\tfiles_create_reflog,\n+\tfiles_delete_reflog,\n+\tfiles_reflog_expire,\n+\n \tfiles_pack_refs,\n \tfiles_peel_ref,\n \tfiles_create_symref,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex cb949c5..ee2bea6 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -211,6 +211,25 @@ int rename_ref_available(const char *oldname, const char *newname);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n+/* reflog functions */\n+typedef int for_each_reflog_ent_fn(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_reflog_ent_reverse_fn(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data);\n+typedef int for_each_reflog_fn(each_ref_fn fn, void *cb_data);\n+typedef int reflog_exists_fn(const char *refname);\n+typedef int create_reflog_fn(const char *refname, int force_create,\n+\t\t\t struct strbuf *err);\n+typedef int delete_reflog_fn(const char *refname);\n+typedef int reflog_expire_fn(const char *refname, const unsigned char *sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data);\n+\n /* misc methods */\n typedef int pack_refs_fn(unsigned int flags);\n typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n@@ -249,6 +268,14 @@ struct ref_storage_be {\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n \n+\tfor_each_reflog_ent_fn *for_each_reflog_ent;\n+\tfor_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;\n+\tfor_each_reflog_fn *for_each_reflog;\n+\treflog_exists_fn *reflog_exists;\n+\tcreate_reflog_fn *create_reflog;\n+\tdelete_reflog_fn *delete_reflog;\n+\treflog_expire_fn *reflog_expire;\n+\n \tpack_refs_fn *pack_refs;\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277569","messageId":"1454701462-3817-7-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 06/21] refs: add method for initial ref transaction commit","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:07Z","receivedAt":"2016-02-05T19:44:07Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 6 ++++++\n refs/files-backend.c | 5 +++--\n refs/refs-internal.h | 1 +\n 3 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 3254378..d481a94 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1258,3 +1258,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t\t\t prepare_fn, should_prune_fn,\n \t\t\t\t\t cleanup_fn, policy_cb_data);\n }\n+\n+int initial_ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->initial_transaction_commit(transaction, err);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b3372e6..723127e 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3337,8 +3337,8 @@ static int ref_present(const char *refname,\n \treturn string_list_has_string(affected_refnames, refname);\n }\n \n-int initial_ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t\t struct strbuf *err)\n+static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3562,6 +3562,7 @@ struct ref_storage_be refs_be_files = {\n \tNULL,\n \t\"files\",\n \tfiles_transaction_commit,\n+\tfiles_initial_transaction_commit,\n \n \tfiles_for_each_reflog_ent,\n \tfiles_for_each_reflog_ent_reverse,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex ee2bea6..142c663 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -267,6 +267,7 @@ struct ref_storage_be {\n \tstruct ref_storage_be *next;\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n+\tref_transaction_commit_fn *initial_transaction_commit;\n \n \tfor_each_reflog_ent_fn *for_each_reflog_ent;\n \tfor_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277587","messageId":"1454701462-3817-8-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 07/21] refs: add method for delete_refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:08Z","receivedAt":"2016-02-05T19:44:08Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In the file-based backend, delete_refs has some special optimization\nto deal with packed refs. In other backends, we might be able to make\nref deletion faster by putting all deletions into a single\ntransaction. So we need a special backend function for this.\n\nSigned-off-by: David Turner \n---\n refs.c | 5 +++++\n refs/files-backend.c | 3 ++-\n refs/refs-internal.h | 2 ++\n 3 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex d481a94..5fbe794 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1123,6 +1123,11 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn the_refs_backend->transaction_commit(transaction, err);\n }\n \n+int delete_refs(struct string_list *refnames)\n+{\n+\treturn the_refs_backend->delete_refs(refnames);\n+}\n+\n const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags)\n {\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 723127e..e86849c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2380,7 +2380,7 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n \treturn 0;\n }\n \n-int delete_refs(struct string_list *refnames)\n+static int files_delete_refs(struct string_list *refnames)\n {\n \tstruct strbuf err = STRBUF_INIT;\n \tint i, result = 0;\n@@ -3575,6 +3575,7 @@ struct ref_storage_be refs_be_files = {\n \tfiles_pack_refs,\n \tfiles_peel_ref,\n \tfiles_create_symref,\n+\tfiles_delete_refs,\n \n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 142c663..1d9e5e0 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -236,6 +236,7 @@ typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n typedef int create_symref_fn(const char *ref_target,\n \t\t\t const char *refs_heads_master,\n \t\t\t const char *logmsg);\n+typedef int delete_refs_fn(struct string_list *refnames);\n \n /* resolution methods */\n typedef const char *resolve_ref_unsafe_fn(const char *ref,\n@@ -280,6 +281,7 @@ struct ref_storage_be {\n \tpack_refs_fn *pack_refs;\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n+\tdelete_refs_fn *delete_refs;\n \n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277572","messageId":"1454701462-3817-9-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 08/21] refs: add methods to init refs db","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:09Z","receivedAt":"2016-02-05T19:44:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might not need the refs/heads directory and so\non, so we make ref db initialization part of the backend.\n\nSigned-off-by: David Turner \n---\n builtin/init-db.c | 20 ++++++++++----------\n refs.c | 5 +++++\n refs.h | 2 ++\n refs/files-backend.c | 16 ++++++++++++++++\n refs/refs-internal.h | 2 ++\n 5 files changed, 35 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 26e1cc3..801e977 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -178,13 +178,7 @@ static int create_default_files(const char *template_path)\n \tchar junk[2];\n \tint reinit;\n \tint filemode;\n-\n-\t/*\n-\t * Create .git/refs/{heads,tags}\n-\t */\n-\tsafe_create_dir(git_path_buf(&buf, \"refs\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/heads\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/tags\"), 1);\n+\tstruct strbuf err = STRBUF_INIT;\n \n \t/* Just look for `init.templatedir` */\n \tgit_config(git_init_db_config, NULL);\n@@ -208,12 +202,18 @@ static int create_default_files(const char *template_path)\n \t */\n \tif (shared_repository) {\n \t\tadjust_shared_perm(get_git_dir());\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/heads\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/tags\"));\n \t}\n \n \t/*\n+\t * We need to create a \"refs\" dir in any case so that older\n+\t * versions of git can tell that this is a repository.\n+\t */\n+\tsafe_create_dir(git_path(\"refs\"), 1);\n+\t\n+\tif (refs_init_db(&err, shared_repository))\n+\t\tdie(\"failed to set up refs db: %s\", err.buf);\n+\n+\t/*\n \t * Create the default symlink from \".git/HEAD\" to the \"master\"\n \t * branch, if it does not exist yet.\n \t */\ndiff --git a/refs.c b/refs.c\nindex 5fbe794..7758bdc 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1117,6 +1117,11 @@ int rename_ref_available(const char *oldname, const char *newname)\n }\n \n /* backend functions */\n+int refs_init_db(struct strbuf *err, int shared)\n+{\n+\treturn the_refs_backend->init_db(err, shared);\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\ndiff --git a/refs.h b/refs.h\nindex 5bc3fbc..c5ecc52 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -66,6 +66,8 @@ extern int ref_exists(const char *refname);\n \n extern int is_branch(const char *refname);\n \n+extern int refs_init_db(struct strbuf *err, int shared);\n+\n /*\n * If refname is a non-symbolic reference that refers to a tag object,\n * and the tag can be (recursively) dereferenced to a non-tag object,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e86849c..45da091 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3558,9 +3558,25 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \treturn -1;\n }\n \n+static int files_init_db(struct strbuf *err, int shared)\n+{\n+\t/*\n+\t * Create .git/refs/{heads,tags}\n+\t */\n+\tsafe_create_dir(git_path(\"refs/heads\"), 1);\n+\tsafe_create_dir(git_path(\"refs/tags\"), 1);\n+\tif (shared) {\n+\t\tadjust_shared_perm(git_path(\"refs\"));\n+\t\tadjust_shared_perm(git_path(\"refs/heads\"));\n+\t\tadjust_shared_perm(git_path(\"refs/tags\"));\n+\t}\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \tNULL,\n \t\"files\",\n+\tfiles_init_db,\n \tfiles_transaction_commit,\n \tfiles_initial_transaction_commit,\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 1d9e5e0..5768fee 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -208,6 +208,7 @@ const char *find_descendant_ref(const char *dirname,\n int rename_ref_available(const char *oldname, const char *newname);\n \n /* refs backends */\n+typedef int ref_init_db_fn(struct strbuf *err, int shared);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n@@ -267,6 +268,7 @@ typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n struct ref_storage_be {\n \tstruct ref_storage_be *next;\n \tconst char *name;\n+\tref_init_db_fn *init_db;\n \tref_transaction_commit_fn *transaction_commit;\n \tref_transaction_commit_fn *initial_transaction_commit;\n \n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277573","messageId":"1454701462-3817-10-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 09/21] refs: add method to rename refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:10Z","receivedAt":"2016-02-05T19:44:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: David Turner \n---\n refs.c | 5 +++++\n refs/files-backend.c | 4 +++-\n refs/refs-internal.h | 9 +++++++++\n 3 files changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 7758bdc..e04fddc 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1133,6 +1133,11 @@ int delete_refs(struct string_list *refnames)\n \treturn the_refs_backend->delete_refs(refnames);\n }\n \n+int rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\treturn the_refs_backend->rename_ref(oldref, newref, logmsg);\n+}\n+\n const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags)\n {\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 45da091..685fc6f 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2502,7 +2502,8 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t const unsigned char *sha1, const char *logmsg,\n \t\t\t int flags, struct strbuf *err);\n \n-int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)\n+static int files_rename_ref(const char *oldrefname, const char *newrefname,\n+\t\t\t const char *logmsg)\n {\n \tunsigned char sha1[20], orig_sha1[20];\n \tint flag = 0, logmoved = 0;\n@@ -3592,6 +3593,7 @@ struct ref_storage_be refs_be_files = {\n \tfiles_peel_ref,\n \tfiles_create_symref,\n \tfiles_delete_refs,\n+\tfiles_rename_ref,\n \n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 5768fee..15fa99c 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -67,6 +67,13 @@ int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n \t\t\t\t each_ref_fn fn, int trim, int flags,\n \t\t\t\t void *cb_data);\n \n+/*\n+ * Check if the new name does not conflict with any existing refs\n+ * (other than possibly the old ref). Return 0 if the ref can be\n+ * renamed to the new name.\n+ */\n+int rename_ref_available(const char *oldname, const char *newname);\n+\n enum peel_status {\n \t/* object was peeled successfully: */\n \tPEEL_PEELED = 0,\n@@ -246,6 +253,7 @@ typedef const char *resolve_ref_unsafe_fn(const char *ref,\n typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);\n typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n+typedef int rename_ref_fn(const char *oldref, const char *newref, const char *logmsg);\n \n /* iteration methods */\n typedef int head_ref_fn(each_ref_fn fn, void *cb_data);\n@@ -284,6 +292,7 @@ struct ref_storage_be {\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n \tdelete_refs_fn *delete_refs;\n+\trename_ref_fn *rename_ref;\n \n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277583","messageId":"1454701462-3817-11-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 10/21] refs: make lock generic","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:11Z","receivedAt":"2016-02-05T19:44:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Instead of using a files-backend-specific struct ref_lock, the generic\nref_transaction struct should provide a void pointer that backends can use\nfor their own lock data.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 29 ++++++++++++++++-------------\n refs/refs-internal.h | 2 +-\n 2 files changed, 17 insertions(+), 14 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 685fc6f..aa11ba2 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3208,11 +3208,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t */\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n+\t\tstruct ref_lock *lock;\n \n \t\tif ((update->flags & REF_HAVE_NEW) &&\n \t\t is_null_sha1(update->new_sha1))\n \t\t\tupdate->flags |= REF_DELETING;\n-\t\tupdate->lock = lock_ref_sha1_basic(\n+\t\tlock = lock_ref_sha1_basic(\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n@@ -3220,7 +3221,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n \t\t\t\terr);\n-\t\tif (!update->lock) {\n+\t\tupdate->backend_data = lock;\n+\t\tif (!lock) {\n \t\t\tchar *reason;\n \n \t\t\tret = (errno == ENOTDIR)\n@@ -3238,12 +3240,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t\t (update->flags & REF_NODEREF));\n \n \t\t\tif (!overwriting_symref &&\n-\t\t\t !hashcmp(update->lock->old_oid.hash, update->new_sha1)) {\n+\t\t\t !hashcmp(lock->old_oid.hash, update->new_sha1)) {\n \t\t\t\t/*\n \t\t\t\t * The reference already has the desired\n \t\t\t\t * value, so we don't need to write it.\n \t\t\t\t */\n-\t\t\t} else if (write_ref_to_lockfile(update->lock,\n+\t\t\t} else if (write_ref_to_lockfile(lock,\n \t\t\t\t\t\t\t update->new_sha1,\n \t\t\t\t\t\t\t err)) {\n \t\t\t\tchar *write_err = strbuf_detach(err, NULL);\n@@ -3252,7 +3254,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t * The lock was freed upon failure of\n \t\t\t\t * write_ref_to_lockfile():\n \t\t\t\t */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t\tstrbuf_addf(err,\n \t\t\t\t\t \"cannot update the ref '%s': %s\",\n \t\t\t\t\t update->refname, write_err);\n@@ -3268,7 +3270,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t * We didn't have to write anything to the lockfile.\n \t\t\t * Close it to free up the file descriptor:\n \t\t\t */\n-\t\t\tif (close_ref(update->lock)) {\n+\t\t\tif (close_ref(lock)) {\n \t\t\t\tstrbuf_addf(err, \"Couldn't close %s.lock\",\n \t\t\t\t\t update->refname);\n \t\t\t\tgoto cleanup;\n@@ -3281,16 +3283,16 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \n \t\tif (update->flags & REF_NEEDS_COMMIT) {\n-\t\t\tif (commit_ref_update(update->lock,\n+\t\t\tif (commit_ref_update(update->backend_data,\n \t\t\t\t\t update->new_sha1, update->msg,\n \t\t\t\t\t update->flags, err)) {\n \t\t\t\t/* freed by commit_ref_update(): */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\n \t\t\t} else {\n \t\t\t\t/* freed by commit_ref_update(): */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t}\n \t\t}\n \t}\n@@ -3298,16 +3300,17 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t/* Perform deletes now that updates are safely completed */\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n+\t\tstruct ref_lock *lock = update->backend_data;\n \n \t\tif (update->flags & REF_DELETING) {\n-\t\t\tif (delete_ref_loose(update->lock, update->type, err)) {\n+\t\t\tif (delete_ref_loose(lock, update->type, err)) {\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (!(update->flags & REF_ISPRUNING))\n \t\t\t\tstring_list_append(&refs_to_delete,\n-\t\t\t\t\t\t update->lock->ref_name);\n+\t\t\t\t\t\t lock->ref_name);\n \t\t}\n \t}\n \n@@ -3323,8 +3326,8 @@ cleanup:\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \n \tfor (i = 0; i < n; i++)\n-\t\tif (updates[i]->lock)\n-\t\t\tunlock_ref(updates[i]->lock);\n+\t\tif (updates[i]->backend_data)\n+\t\t\tunlock_ref(updates[i]->backend_data);\n \tstring_list_clear(&refs_to_delete, 0);\n \tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 15fa99c..3dec075 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -161,7 +161,7 @@ struct ref_update {\n \t * REF_DELETING, and REF_ISPRUNING:\n \t */\n \tunsigned int flags;\n-\tstruct ref_lock *lock;\n+\tvoid *backend_data;\n \tint type;\n \tchar *msg;\n \tconst char refname[FLEX_ARRAY];\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277582","messageId":"1454701462-3817-12-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 11/21] refs: move duplicate check to common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:12Z","receivedAt":"2016-02-05T19:44:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The check for duplicate refnames in a transaction is needed for\nall backends, so move it to the common code.\n\nref_transaction_commit_fn gains a new argument, the sorted\nstring_list of affected refnames.\n\nSigned-off-by: David Turner \n---\n refs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n refs/files-backend.c | 57 ++++---------------------------------------\n refs/refs-internal.h | 1 +\n 3 files changed, 73 insertions(+), 54 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex e04fddc..283a5ec 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1116,6 +1116,36 @@ int rename_ref_available(const char *oldname, const char *newname)\n \treturn ret;\n }\n \n+/*\n+ * Return 1 if there are any duplicate refnames in the updates in\n+ * `transaction`, and fill in err with an appropriate error message.\n+ * Fill in `refnames` with the refnames from the transaction.\n+ */\n+static int get_affected_refnames(struct ref_transaction *transaction,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint i, n = transaction->nr;\n+\tstruct ref_update **updates;\n+\n+\tassert(err);\n+\n+\tupdates = transaction->updates;\n+\t/* Fail if a refname appears more than once in the transaction: */\n+\tfor (i = 0; i < n; i++)\n+\t\tstring_list_append(refnames, updates[i]->refname);\n+\tstring_list_sort(refnames);\n+\n+\tfor (i = 1; i < n; i++)\n+\t\tif (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Multiple updates for ref '%s' not allowed.\",\n+\t\t\t\t refnames->items[i].string);\n+\t\t\treturn 1;\n+\t\t}\n+\treturn 0;\n+}\n+\n /* backend functions */\n int refs_init_db(struct strbuf *err, int shared)\n {\n@@ -1125,7 +1155,29 @@ int refs_init_db(struct strbuf *err, int shared)\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n-\treturn the_refs_backend->transaction_commit(transaction, err);\n+\tint ret = -1;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\n+\tassert(err);\n+\n+\tif (transaction->state != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: commit called for transaction that is not open\");\n+\n+\tif (!transaction->nr) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\treturn 0;\n+\t}\n+\n+\tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = the_refs_backend->transaction_commit(transaction,\n+\t\t\t\t\t\t &affected_refnames, err);\n+done:\n+\tstring_list_clear(&affected_refnames, 0);\n+\treturn ret;\n }\n \n int delete_refs(struct string_list *refnames)\n@@ -1277,5 +1329,18 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n int initial_ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n {\n-\treturn the_refs_backend->initial_transaction_commit(transaction, err);\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tint ret;\n+\n+\tif (get_affected_refnames(transaction,\n+\t\t\t\t &affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\tret = the_refs_backend->initial_transaction_commit(transaction,\n+\t\t\t\t\t\t\t &affected_refnames,\n+\t\t\t\t\t\t\t err);\n+done:\n+\tstring_list_clear(&affected_refnames, 0);\n+\treturn ret;\n }\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa11ba2..0ad60bc 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3154,24 +3154,8 @@ static int files_for_each_reflog(each_ref_fn fn, void *cb_data)\n \treturn retval;\n }\n \n-static int ref_update_reject_duplicates(struct string_list *refnames,\n-\t\t\t\t\tstruct strbuf *err)\n-{\n-\tint i, n = refnames->nr;\n-\n-\tassert(err);\n-\n-\tfor (i = 1; i < n; i++)\n-\t\tif (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {\n-\t\t\tstrbuf_addf(err,\n-\t\t\t\t \"Multiple updates for ref '%s' not allowed.\",\n-\t\t\t\t refnames->items[i].string);\n-\t\t\treturn 1;\n-\t\t}\n-\treturn 0;\n-}\n-\n static int files_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n@@ -3179,26 +3163,6 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \tstruct ref_update **updates = transaction->updates;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_NODUP;\n \tstruct string_list_item *ref_to_delete;\n-\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n-\n-\tassert(err);\n-\n-\tif (transaction->state != REF_TRANSACTION_OPEN)\n-\t\tdie(\"BUG: commit called for transaction that is not open\");\n-\n-\tif (!n) {\n-\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n-\t\treturn 0;\n-\t}\n-\n-\t/* Fail if a refname appears more than once in the transaction: */\n-\tfor (i = 0; i < n; i++)\n-\t\tstring_list_append(&affected_refnames, updates[i]->refname);\n-\tstring_list_sort(&affected_refnames);\n-\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n \n \t/*\n \t * Acquire all locks, verify old values if provided, check\n@@ -3217,7 +3181,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n-\t\t\t\t&affected_refnames, NULL,\n+\t\t\t\taffected_refnames, NULL,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n \t\t\t\terr);\n@@ -3329,7 +3293,6 @@ cleanup:\n \t\tif (updates[i]->backend_data)\n \t\t\tunlock_ref(updates[i]->backend_data);\n \tstring_list_clear(&refs_to_delete, 0);\n-\tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\n }\n \n@@ -3342,27 +3305,18 @@ static int ref_present(const char *refname,\n }\n \n static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n-\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \n \tassert(err);\n \n \tif (transaction->state != REF_TRANSACTION_OPEN)\n \t\tdie(\"BUG: commit called for transaction that is not open\");\n \n-\t/* Fail if a refname appears more than once in the transaction: */\n-\tfor (i = 0; i < n; i++)\n-\t\tstring_list_append(&affected_refnames, updates[i]->refname);\n-\tstring_list_sort(&affected_refnames);\n-\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n-\n \t/*\n \t * It's really undefined to call this function in an active\n \t * repository or when there are existing references: we are\n@@ -3375,7 +3329,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \t * so here we really only check that none of the references\n \t * that we are creating already exists.\n \t */\n-\tif (for_each_rawref(ref_present, &affected_refnames))\n+\tif (for_each_rawref(ref_present, affected_refnames))\n \t\tdie(\"BUG: initial ref transaction called with existing refs\");\n \n \tfor (i = 0; i < n; i++) {\n@@ -3385,7 +3339,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \t\t !is_null_sha1(update->old_sha1))\n \t\t\tdie(\"BUG: initial ref transaction with old_sha1 set\");\n \t\tif (verify_refname_available(update->refname,\n-\t\t\t\t\t &affected_refnames, NULL,\n+\t\t\t\t\t affected_refnames, NULL,\n \t\t\t\t\t err)) {\n \t\t\tret = TRANSACTION_NAME_CONFLICT;\n \t\t\tgoto cleanup;\n@@ -3416,7 +3370,6 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \n cleanup:\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n-\tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\n }\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 3dec075..fc5d1db 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -217,6 +217,7 @@ int rename_ref_available(const char *oldname, const char *newname);\n /* refs backends */\n typedef int ref_init_db_fn(struct strbuf *err, int shared);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t struct strbuf *err);\n \n /* reflog functions */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277575","messageId":"1454701462-3817-13-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 12/21] refs: allow log-only updates","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:13Z","receivedAt":"2016-02-05T19:44:13Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The refs infrastructure learns about log-only ref updates, which only\nupdate the reflog. Later, we will use this to separate symbolic\nreference resolution from ref updating.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 15 ++++++++++-----\n refs/refs-internal.h | 2 ++\n 2 files changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0ad60bc..0fdcdc7 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2845,7 +2845,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t}\n \t\t}\n \t}\n-\tif (commit_ref(lock)) {\n+\tif (!(flags & REF_LOG_ONLY) && commit_ref(lock)) {\n \t\terror(\"Couldn't set %s\", lock->ref_name);\n \t\tunlock_ref(lock);\n \t\treturn -1;\n@@ -3199,7 +3199,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\tgoto cleanup;\n \t\t}\n \t\tif ((update->flags & REF_HAVE_NEW) &&\n-\t\t !(update->flags & REF_DELETING)) {\n+\t\t !(update->flags & REF_DELETING) &&\n+\t\t !(update->flags & REF_LOG_ONLY)) {\n \t\t\tint overwriting_symref = ((update->type & REF_ISSYMREF) &&\n \t\t\t\t\t\t (update->flags & REF_NODEREF));\n \n@@ -3229,7 +3230,9 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->flags |= REF_NEEDS_COMMIT;\n \t\t\t}\n \t\t}\n-\t\tif (!(update->flags & REF_NEEDS_COMMIT)) {\n+\n+\t\tif (!(update->flags & REF_LOG_ONLY) &&\n+\t\t !(update->flags & REF_NEEDS_COMMIT)) {\n \t\t\t/*\n \t\t\t * We didn't have to write anything to the lockfile.\n \t\t\t * Close it to free up the file descriptor:\n@@ -3246,7 +3249,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n-\t\tif (update->flags & REF_NEEDS_COMMIT) {\n+\t\tif (update->flags & REF_NEEDS_COMMIT ||\n+\t\t update->flags & REF_LOG_ONLY) {\n \t\t\tif (commit_ref_update(update->backend_data,\n \t\t\t\t\t update->new_sha1, update->msg,\n \t\t\t\t\t update->flags, err)) {\n@@ -3266,7 +3270,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \t\tstruct ref_lock *lock = update->backend_data;\n \n-\t\tif (update->flags & REF_DELETING) {\n+\t\tif (update->flags & REF_DELETING &&\n+\t\t !(update->flags & REF_LOG_ONLY)) {\n \t\t\tif (delete_ref_loose(lock, update->type, err)) {\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fc5d1db..b5d0ab8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -42,6 +42,8 @@\n * value to ref_update::flags\n */\n \n+#define REF_LOG_ONLY 0x80\n+\n /*\n * Return true iff refname is minimally safe. \"Safe\" here means that\n * deleting a loose reference by this name will not do any damage, for\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277574","messageId":"1454701462-3817-14-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 13/21] refs: resolve symbolic refs first","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:14Z","receivedAt":"2016-02-05T19:44:14Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Before committing ref updates, split symbolic ref updates into two\nparts: an update to the underlying ref, and a log-only update to the\nsymbolic ref. This ensures that both references are locked correctly\nwhile their reflogs are updated.\n\nIt is still possible to confuse git by concurrent updates, since the\nsplitting of symbolic refs does not happen under lock. So a symbolic ref\ncould be replaced by a plain ref in the middle of this operation, which\nwould lead to reflog discontinuities and missed old-ref checks.\n\nSigned-off-by: David Turner \n---\n refs.c | 69 +++++++++++++++++++++++++++\n refs/files-backend.c | 132 ++++++++++++++++++++++++++-------------------------\n refs/refs-internal.h | 8 ++++\n 3 files changed, 145 insertions(+), 64 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 283a5ec..227c018 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1152,6 +1152,71 @@ int refs_init_db(struct strbuf *err, int shared)\n \treturn the_refs_backend->init_db(err, shared);\n }\n \n+/*\n+ * Special case for symbolic refs when REF_NODEREF is not turned on.\n+ * Dereference them here, mark them REF_LOG_ONLY, and add an update\n+ * for the underlying ref.\n+ */\n+static int dereference_symrefs(struct ref_transaction *transaction,\n+\t\t\t struct strbuf *err)\n+{\n+\tint i;\n+\tint nr = transaction->nr;\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tconst char *resolved;\n+\t\tunsigned char sha1[20];\n+\t\tint resolve_flags = 0;\n+\t\tint mustexist = update->flags & REF_HAVE_OLD &&\n+\t\t\t!is_null_sha1(update->old_sha1);\n+\t\tint deleting = (update->flags & REF_HAVE_NEW) &&\n+\t\t\tis_null_sha1(update->new_sha1);\n+\n+\t\tif (mustexist)\n+\t\t\tresolve_flags |= RESOLVE_REF_READING;\n+\t\tif (deleting)\n+\t\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME |\n+\t\t\t\tRESOLVE_REF_NO_RECURSE;\n+\n+\t\tif (strcmp(update->refname, \"HEAD\"))\n+\t\t\tupdate->flags |= REF_IS_NOT_HEAD;\n+\n+\t\tresolved = resolve_ref_unsafe(update->refname, resolve_flags,\n+\t\t\t\t\t sha1, &update->type);\n+\t\tif (!resolved) {\n+\t\t\t/*\n+\t\t\t * We may notice this breakage later and die\n+\t\t\t * with a sensible error message\n+\t\t\t */\n+\t\t\tupdate->type |= REF_ISBROKEN;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\thashcpy(update->read_sha1, sha1);\n+\n+\t\tif (update->flags & REF_NODEREF ||\n+\t\t !(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\t/* Create a new transaction for the underlying ref */\n+\t\tif (ref_transaction_update(transaction,\n+\t\t\t\t\t resolved,\n+\t\t\t\t\t update->new_sha1,\n+\t\t\t\t\t (update->flags & REF_HAVE_OLD) ?\n+\t\t\t\t\t update->old_sha1 : NULL,\n+\t\t\t\t\t update->flags & ~REF_IS_NOT_HEAD,\n+\t\t\t\t\t update->msg, err))\n+\t\t\treturn -1;\n+\n+\t\t/* Make the symbolic ref update non-recursive */\n+\t\tupdate->flags |= REF_LOG_ONLY | REF_NODEREF;\n+\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n@@ -1168,6 +1233,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\treturn 0;\n \t}\n \n+\tret = dereference_symrefs(transaction, err);\n+\tif (ret)\n+\t\tgoto done;\n+\n \tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto done;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0fdcdc7..d4f9040 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -7,7 +7,6 @@\n \n struct ref_lock {\n \tchar *ref_name;\n-\tchar *orig_ref_name;\n \tstruct lock_file *lk;\n \tstruct object_id old_oid;\n };\n@@ -1857,7 +1856,6 @@ static void unlock_ref(struct ref_lock *lock)\n \tif (lock->lk)\n \t\trollback_lock_file(lock->lk);\n \tfree(lock->ref_name);\n-\tfree(lock->orig_ref_name);\n \tfree(lock);\n }\n \n@@ -1913,6 +1911,7 @@ static int remove_empty_directories(struct strbuf *path)\n */\n static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t\t const unsigned char *read_sha1,\n \t\t\t\t\t const struct string_list *extras,\n \t\t\t\t\t const struct string_list *skip,\n \t\t\t\t\t unsigned int flags, int *type_p,\n@@ -1920,14 +1919,14 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n {\n \tstruct strbuf ref_file = STRBUF_INIT;\n \tstruct strbuf orig_ref_file = STRBUF_INIT;\n-\tconst char *orig_refname = refname;\n \tstruct ref_lock *lock;\n \tint last_errno = 0;\n \tint type;\n \tint lflags = 0;\n \tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n-\tint resolve_flags = 0;\n+\tint resolve_flags = RESOLVE_REF_NO_RECURSE;\n \tint attempts_remaining = 3;\n+\tint resolved;\n \n \tassert(err);\n \n@@ -1937,65 +1936,65 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\tresolve_flags |= RESOLVE_REF_READING;\n \tif (flags & REF_DELETING)\n \t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n-\tif (flags & REF_NODEREF) {\n-\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n+\tif (flags & REF_NODEREF)\n \t\tlflags |= LOCK_NO_DEREF;\n-\t}\n \n-\trefname = resolve_ref_unsafe(refname, resolve_flags,\n-\t\t\t\t lock->old_oid.hash, &type);\n-\tif (!refname && errno == EISDIR) {\n-\t\t/*\n-\t\t * we are trying to lock foo but we used to\n-\t\t * have foo/bar which now does not exist;\n-\t\t * it is normal for the empty directory 'foo'\n-\t\t * to remain.\n-\t\t */\n-\t\tstrbuf_git_path(&orig_ref_file, \"%s\", orig_refname);\n-\t\tif (remove_empty_directories(&orig_ref_file)) {\n+\tif (type_p && *type_p & REF_ISSYMREF) {\n+\t\thashcpy(lock->old_oid.hash, read_sha1);\n+\t} else {\n+\t\tresolved = !!resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t\tlock->old_oid.hash, &type);\n+\t\tif (!resolved && errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * we are trying to lock foo but we used to\n+\t\t\t * have foo/bar which now does not exist;\n+\t\t\t * it is normal for the empty directory 'foo'\n+\t\t\t * to remain.\n+\t\t\t */\n+\t\t\tstrbuf_git_path(&orig_ref_file, \"%s\", refname);\n+\t\t\tif (remove_empty_directories(&orig_ref_file)) {\n+\t\t\t\tstruct ref_dir *loose_refs;\n+\t\t\t\tloose_refs = get_loose_refs(&ref_cache);\n+\t\t\t\tlast_errno = errno;\n+\t\t\t\tif (!verify_refname_available_dir(refname, extras, skip,\n+\t\t\t\t\t\t\t\t loose_refs, err))\n+\t\t\t\t\tstrbuf_addf(err, \"there are still refs under '%s'\",\n+\t\t\t\t\t\t refname);\n+\t\t\t\tgoto error_return;\n+\t\t\t}\n+\t\t\tresolved = !!resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t\t\tlock->old_oid.hash, &type);\n+\t\t}\n+\n+\t\tif (type_p)\n+\t\t\t*type_p = type;\n+\t\tif (!resolved) {\n \t\t\tlast_errno = errno;\n-\t\t\tif (!verify_refname_available_dir(orig_refname, extras, skip,\n+\t\t\tif (last_errno != ENOTDIR ||\n+\t\t\t !verify_refname_available_dir(refname, extras, skip,\n \t\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n-\t\t\t\tstrbuf_addf(err, \"there are still refs under '%s'\",\n-\t\t\t\t\t orig_refname);\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t \"unable to resolve reference %s: %s\",\n+\t\t\t\t\t refname, strerror(last_errno));\n+\n+\t\t\tgoto error_return;\n+\t\t}\n+\t\t/*\n+\t\t * If the ref did not exist and we are creating it, make sure\n+\t\t * there is no existing packed ref whose name begins with our\n+\t\t * refname, nor a packed ref whose name is a proper prefix of\n+\t\t * our refname.\n+\t\t */\n+\t\tif (is_null_oid(&lock->old_oid) &&\n+\t\t verify_refname_available_dir(refname, extras, skip,\n+\t\t\t\t\t\t get_packed_refs(&ref_cache), err)) {\n+\t\t\tlast_errno = ENOTDIR;\n \t\t\tgoto error_return;\n \t\t}\n-\t\trefname = resolve_ref_unsafe(orig_refname, resolve_flags,\n-\t\t\t\t\t lock->old_oid.hash, &type);\n-\t}\n-\tif (type_p)\n-\t *type_p = type;\n-\tif (!refname) {\n-\t\tlast_errno = errno;\n-\t\tif (last_errno != ENOTDIR ||\n-\t\t !verify_refname_available_dir(orig_refname, extras, skip,\n-\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n-\t\t\tstrbuf_addf(err, \"unable to resolve reference %s: %s\",\n-\t\t\t\t orig_refname, strerror(last_errno));\n-\n-\t\tgoto error_return;\n-\t}\n-\n-\tif (flags & REF_NODEREF)\n-\t\trefname = orig_refname;\n-\n-\t/*\n-\t * If the ref did not exist and we are creating it, make sure\n-\t * there is no existing packed ref whose name begins with our\n-\t * refname, nor a packed ref whose name is a proper prefix of\n-\t * our refname.\n-\t */\n-\tif (is_null_oid(&lock->old_oid) &&\n-\t verify_refname_available_dir(refname, extras, skip,\n-\t\t\t\t\t get_packed_refs(&ref_cache), err)) {\n-\t\tlast_errno = ENOTDIR;\n-\t\tgoto error_return;\n \t}\n-\n \tlock->lk = xcalloc(1, sizeof(struct lock_file));\n \n \tlock->ref_name = xstrdup(refname);\n-\tlock->orig_ref_name = xstrdup(orig_refname);\n \tstrbuf_git_path(&ref_file, \"%s\", refname);\n \n retry:\n@@ -2027,7 +2026,13 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\t\tgoto error_return;\n \t\t}\n \t}\n-\tif (verify_lock(lock, old_sha1, mustexist, err)) {\n+\n+\tif (type_p && *type_p & REF_ISSYMREF && !(*type_p & REF_ISBROKEN)) {\n+\t\t/*\n+\t\t * Old hash verification for symrefs happens on their\n+\t\t * base ref.\n+\t\t */\n+\t} else if (verify_lock(lock, old_sha1, mustexist, err)) {\n \t\tlast_errno = errno;\n \t\tgoto error_return;\n \t}\n@@ -2561,7 +2566,7 @@ static int files_rename_ref(const char *oldrefname, const char *newrefname,\n \n \tlogmoved = log;\n \n-\tlock = lock_ref_sha1_basic(newrefname, NULL, NULL, NULL, 0, NULL, &err);\n+\tlock = lock_ref_sha1_basic(newrefname, NULL, NULL, NULL, NULL, 0, NULL, &err);\n \tif (!lock) {\n \t\terror(\"unable to rename '%s' to '%s': %s\", oldrefname, newrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -2579,7 +2584,7 @@ static int files_rename_ref(const char *oldrefname, const char *newrefname,\n \treturn 0;\n \n rollback:\n-\tlock = lock_ref_sha1_basic(oldrefname, NULL, NULL, NULL, 0, NULL, &err);\n+\tlock = lock_ref_sha1_basic(oldrefname, NULL, NULL, NULL, NULL, 0, NULL, &err);\n \tif (!lock) {\n \t\terror(\"unable to lock %s for rollback: %s\", oldrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -2807,9 +2812,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t int flags, struct strbuf *err)\n {\n \tclear_loose_ref_cache(&ref_cache);\n-\tif (log_ref_write(lock->ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0 ||\n-\t (strcmp(lock->ref_name, lock->orig_ref_name) &&\n-\t log_ref_write(lock->orig_ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0)) {\n+\tif (log_ref_write(lock->ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \t\tstrbuf_addf(err, \"Cannot update the ref '%s': %s\",\n \t\t\t lock->ref_name, old_msg);\n@@ -2817,7 +2820,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\tunlock_ref(lock);\n \t\treturn -1;\n \t}\n-\tif (strcmp(lock->orig_ref_name, \"HEAD\") != 0) {\n+\tif (flags & REF_IS_NOT_HEAD) {\n \t\t/*\n \t\t * Special hack: If a branch is updated directly and HEAD\n \t\t * points to it (may happen on the remote side of a push\n@@ -2912,8 +2915,8 @@ static int files_create_symref(const char *refname,\n \tstruct ref_lock *lock;\n \tint ret;\n \n-\tlock = lock_ref_sha1_basic(refname, NULL, NULL, NULL, REF_NODEREF, NULL,\n-\t\t\t\t &err);\n+\tlock = lock_ref_sha1_basic(refname, NULL, NULL, NULL, NULL, REF_NODEREF,\n+\t\t\t\t NULL, &err);\n \tif (!lock) {\n \t\terror(\"%s\", err.buf);\n \t\tstrbuf_release(&err);\n@@ -3181,6 +3184,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n+\t\t\t\tupdate->read_sha1,\n \t\t\t\taffected_refnames, NULL,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n@@ -3427,7 +3431,7 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \tstruct ref_lock *lock;\n \tchar *log_file;\n \tint status = 0;\n-\tint type;\n+\tint type = 0;\n \tstruct strbuf err = STRBUF_INIT;\n \n \tmemset(&cb, 0, sizeof(cb));\n@@ -3440,7 +3444,7 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \t * reference itself, plus we might need to update the\n \t * reference if --updateref was specified:\n \t */\n-\tlock = lock_ref_sha1_basic(refname, sha1, NULL, NULL, 0, &type, &err);\n+\tlock = lock_ref_sha1_basic(refname, sha1, NULL, NULL, NULL, 0, &type, &err);\n \tif (!lock) {\n \t\terror(\"cannot lock ref '%s': %s\", refname, err.buf);\n \t\tstrbuf_release(&err);\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex b5d0ab8..75b4389 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -44,6 +44,8 @@\n \n #define REF_LOG_ONLY 0x80\n \n+#define REF_IS_NOT_HEAD 0x100\n+\n /*\n * Return true iff refname is minimally safe. \"Safe\" here means that\n * deleting a loose reference by this name will not do any damage, for\n@@ -159,6 +161,12 @@ struct ref_update {\n \t */\n \tunsigned char old_sha1[20];\n \t/*\n+\t * During the symbolic ref split stage, we resolve refs.\n+\t * We'll re-resolve non-symbolic refs once they are locked,\n+\t * but we store this to avoid re-resolving symbolic refs.\n+\t */\n+\tunsigned char read_sha1[20];\n+\t/*\n \t * One or more of REF_HAVE_NEW, REF_HAVE_OLD, REF_NODEREF,\n \t * REF_DELETING, and REF_ISPRUNING:\n \t */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277576","messageId":"1454701462-3817-15-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 14/21] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:15Z","receivedAt":"2016-02-05T19:44:15Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Always handle non-normal (per-worktree or pseudo) refs in the files\nbackend instead of alternate backends.\n\nSometimes a ref transaction will update both a per-worktree ref and a\nnormal ref. For instance, an ordinary commit might update\nrefs/heads/master and HEAD (or at least HEAD's reflog).\n\nUpdates to normal refs continue to go through the chosen backend.\n\nUpdates to non-normal refs are moved to a separate files backend\ntransaction.\n\nSigned-off-by: David Turner \n---\n refs.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 79 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 227c018..18ba356 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -9,6 +9,11 @@\n #include \"object.h\"\n #include \"tag.h\"\n \n+static const char split_transaction_fail_warning[] = N_(\n+\t\"A ref transaction was split across two refs backends. Part of the \"\n+\t\"transaction succeeded, but then the update to the per-worktree refs \"\n+\t\"failed. Your repository may be in an inconsistent state.\");\n+\n /*\n * We always have a files backend and it is the default.\n */\n@@ -791,6 +796,13 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \tfree(transaction);\n }\n \n+static void add_update_obj(struct ref_transaction *transaction,\n+\t\t\t struct ref_update *update)\n+{\n+\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n+\ttransaction->updates[transaction->nr++] = update;\n+}\n+\n static struct ref_update *add_update(struct ref_transaction *transaction,\n \t\t\t\t const char *refname)\n {\n@@ -798,8 +810,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \tstruct ref_update *update = xcalloc(1, sizeof(*update) + len);\n \n \tmemcpy((char *)update->refname, refname, len); /* includes NUL */\n-\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n-\ttransaction->updates[transaction->nr++] = update;\n+\tadd_update_obj(transaction, update);\n \treturn update;\n }\n \n@@ -1217,11 +1228,38 @@ static int dereference_symrefs(struct ref_transaction *transaction,\n \treturn 0;\n }\n \n+/*\n+ * Move all non-normal ref updates into a specially-created\n+ * files-backend transaction\n+ */\n+static int move_abnormal_ref_updates(struct ref_transaction *transaction,\n+\t\t\t\t struct ref_transaction *files_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tint last;\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\n+\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n+\t\t\tcontinue;\n+\n+\t\tlast = --transaction->nr;\n+\t\ttransaction->updates[i] = transaction->updates[last];\n+\t\tadd_update_obj(files_transaction, update);\n+\t}\n+\n+\treturn 0;\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n \tint ret = -1;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct ref_transaction *files_transaction = NULL;\n \n \tassert(err);\n \n@@ -1237,6 +1275,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tif (ret)\n \t\tgoto done;\n \n+\tif (the_refs_backend != &refs_be_files) {\n+\t\tfiles_transaction = ref_transaction_begin(err);\n+\t\tif (!files_transaction)\n+\t\t\tgoto done;\n+\n+\t\tret = move_abnormal_ref_updates(transaction, files_transaction,\n+\t\t\t\t\t\terr);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\t/* files backend commit */\n+\t\tif (get_affected_refnames(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err)) {\n+\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/* main backend commit */\n \tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto done;\n@@ -1244,8 +1302,24 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \tret = the_refs_backend->transaction_commit(transaction,\n \t\t\t\t\t\t &affected_refnames, err);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (files_transaction) {\n+\t\tret = refs_be_files.transaction_commit(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err);\n+\t\tif (ret) {\n+\t\t\twarning(split_transaction_fail_warning);\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n done:\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&files_affected_refnames, 0);\n+\tif (files_transaction)\n+\t\tref_transaction_free(files_transaction);\n \treturn ret;\n }\n \n@@ -1285,6 +1359,9 @@ int peel_ref(const char *refname, unsigned char *sha1)\n int create_symref(const char *ref_target, const char *refs_heads_master,\n \t\t const char *logmsg)\n {\n+\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.create_symref(ref_target, refs_heads_master,\n+\t\t\t\t\t\t logmsg);\n \treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n \t\t\t\t\t logmsg);\n }\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277584","messageId":"1454701462-3817-16-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 15/21] init: allow alternate ref strorage to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:16Z","receivedAt":"2016-02-05T19:44:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"git init learns a new argument --ref-storage. Presently, only\n\"files\" is supported, but later we will add other storage backends.\n\nWhen this argument is used, the repository's extensions.refStorage\nconfiguration value is set (as well as core.repositoryformatversion),\nand the ref storage backend's initdb function is used to set up the\nref database.\n\nSigned-off-by: David Turner \nSigned-off-by: SZEDER Gábor \n---\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 7 +++++-\n builtin/init-db.c | 44 +++++++++++++++++++++++++++-------\n cache.h | 2 ++\n contrib/completion/git-completion.bash | 3 ++-\n path.c | 29 ++++++++++++++++++++--\n refs.c | 8 +++++++\n refs.h | 8 +++++++\n setup.c | 12 ++++++++++\n t/t0001-init.sh | 25 +++++++++++++++++++\n 10 files changed, 127 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt\nindex 648a6cd..d03fb69 100644\n--- a/Documentation/git-init-db.txt\n+++ b/Documentation/git-init-db.txt\n@@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n SYNOPSIS\n --------\n [verse]\n-'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]]\n+'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]] [--ref-storage=]\n \n \n DESCRIPTION\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex 8174d27..d2b150f 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git init' [-q | --quiet] [--bare] [--template=]\n \t [--separate-git-dir ]\n \t [--shared[=]] [directory]\n-\n+\t [--ref-storage=]\n \n DESCRIPTION\n -----------\n@@ -113,6 +113,11 @@ does not exist, it will be created.\n \n --\n \n+--ref-storage=::\n+Type of refs storage backend. Default is to use the original \"files\"\n+storage, which stores ref data in files in .git/refs and\n+.git/packed-refs.\n+\n TEMPLATE DIRECTORY\n ------------------\n \ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 801e977..d331ce8 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -24,6 +24,7 @@ static int init_is_bare_repository = 0;\n static int init_shared_repository = -1;\n static const char *init_db_template_dir;\n static const char *git_link;\n+static char *requested_ref_storage_backend;\n \n static void copy_templates_1(struct strbuf *path, struct strbuf *template,\n \t\t\t DIR *dir)\n@@ -179,6 +180,7 @@ static int create_default_files(const char *template_path)\n \tint reinit;\n \tint filemode;\n \tstruct strbuf err = STRBUF_INIT;\n+\tint repo_version = 0;\n \n \t/* Just look for `init.templatedir` */\n \tgit_config(git_init_db_config, NULL);\n@@ -205,6 +207,32 @@ static int create_default_files(const char *template_path)\n \t}\n \n \t/*\n+\t * Create the default symlink from \".git/HEAD\" to the \"master\"\n+\t * branch, if it does not exist yet.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK)\n+\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n+\tif (reinit) {\n+\t\tif (requested_ref_storage_backend &&\n+\t\t strcmp(ref_storage_backend, requested_ref_storage_backend))\n+\t\t\tdie(_(\"You can't change the refs storage type (was %s; you requested %s)\"),\n+\t\t\t ref_storage_backend,\n+\t\t\t requested_ref_storage_backend);\n+\t}\n+\n+\tif (requested_ref_storage_backend)\n+\t\tref_storage_backend = requested_ref_storage_backend;\n+\tif (strcmp(ref_storage_backend, \"files\")) {\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_backend);\n+\t\tgit_config_set(\"core.repositoryformatversion\", ref_storage_backend);\n+\t\tif (set_ref_storage_backend(ref_storage_backend))\n+\t\t\tdie(_(\"Unknown ref storage backend %s\"),\n+\t\t\t ref_storage_backend);\n+\t\trepo_version = 1;\n+\t}\n+\n+\t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n \t */\n@@ -213,13 +241,6 @@ static int create_default_files(const char *template_path)\n \tif (refs_init_db(&err, shared_repository))\n \t\tdie(\"failed to set up refs db: %s\", err.buf);\n \n-\t/*\n-\t * Create the default symlink from \".git/HEAD\" to the \"master\"\n-\t * branch, if it does not exist yet.\n-\t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tif (create_symref(\"HEAD\", \"refs/heads/master\", NULL) < 0)\n \t\t\texit(1);\n@@ -227,7 +248,7 @@ static int create_default_files(const char *template_path)\n \n \t/* This forces creation of new config file */\n \txsnprintf(repo_version_string, sizeof(repo_version_string),\n-\t\t \"%d\", GIT_REPO_VERSION);\n+\t\t \"%d\", repo_version);\n \tgit_config_set(\"core.repositoryformatversion\", repo_version_string);\n \n \t/* Check filemode trustability */\n@@ -475,11 +496,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n+\t\tOPT_STRING(0, \"ref-storage\", &requested_ref_storage_backend,\n+\t\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \t\tOPT_END()\n \t};\n \n \targc = parse_options(argc, argv, prefix, init_db_options, init_db_usage, 0);\n \n+\tif (requested_ref_storage_backend &&\n+\t !ref_storage_backend_exists(requested_ref_storage_backend))\n+\t\tdie(_(\"Unknown ref storage backend %s\"),\n+\t\t requested_ref_storage_backend);\n+\n \tif (real_git_dir && !is_absolute_path(real_git_dir))\n \t\treal_git_dir = xstrdup(real_path(real_git_dir));\n \ndiff --git a/cache.h b/cache.h\nindex 4307174..a54c97b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -734,6 +734,8 @@ extern enum object_creation_mode object_creation_mode;\n \n extern char *notes_ref_name;\n \n+extern const char *ref_storage_backend;\n+\n extern int grafts_replace_parents;\n \n /*\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 45ec47f..0138d03 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -1362,7 +1362,8 @@ _git_init ()\n \t\treturn\n \t\t;;\n \t--*)\n-\t\t__gitcomp \"--quiet --bare --template= --shared --shared=\"\n+\t\t__gitcomp \"--quiet --bare --template= --shared --shared=\n+\t\t\t--ref-storage=\"\n \t\treturn\n \t\t;;\n \tesac\ndiff --git a/path.c b/path.c\nindex 8b7e168..2e67a2b 100644\n--- a/path.c\n+++ b/path.c\n@@ -2,6 +2,7 @@\n * Utilities for paths and pathnames\n */\n #include \"cache.h\"\n+#include \"refs.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"dir.h\"\n@@ -511,8 +512,32 @@ int validate_headref(const char *path)\n \tint fd;\n \tssize_t len;\n \n-\tif (lstat(path, &st) < 0)\n-\t\treturn -1;\n+\tif (lstat(path, &st) < 0) {\n+\t\tint backend_type_set;\n+\t\tstruct strbuf config_path = STRBUF_INIT;\n+\t\tchar *pathdup = xstrdup(path);\n+\t\tchar *git_dir = dirname(pathdup);\n+\t\tchar *storage = NULL;\n+\n+\t\tstrbuf_addf(&config_path, \"%s/%s\", git_dir, \"config\");\n+\t\tfree(pathdup);\n+\n+\t\tif (git_config_from_file(ref_storage_backend_config,\n+\t\t\t\t\t config_path.buf, &storage)) {\n+\t\t\tstrbuf_release(&config_path);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tbackend_type_set = !!storage;\n+\t\tfree((void *)storage);\n+\t\tstrbuf_release(&config_path);\n+\t\t/*\n+\t\t * Alternate backends are assumed to keep HEAD\n+\t\t * in a valid state, so there's no need to do\n+\t\t * further validation.\n+\t\t */\n+\t\treturn backend_type_set ? 0 : -1;\n+\t}\n \n \t/* Make sure it is a \"refs/..\" symlink */\n \tif (S_ISLNK(st.st_mode)) {\ndiff --git a/refs.c b/refs.c\nindex 18ba356..191b0a2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -23,6 +23,14 @@ static struct ref_storage_be *the_refs_backend = &refs_be_files;\n */\n static struct ref_storage_be *refs_backends = &refs_be_files;\n \n+const char *ref_storage_backend = \"files\";\n+\n+void register_ref_storage_backend(struct ref_storage_be *be)\n+{\n+\tbe->next = refs_backends;\n+\trefs_backends = be;\n+}\n+\n int set_ref_storage_backend(const char *name)\n {\n \tstruct ref_storage_be *be;\ndiff --git a/refs.h b/refs.h\nindex c5ecc52..680a535 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -511,10 +511,18 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t void *policy_cb_data);\n \n /*\n+ * Read the refdb storage backend name out of the config file\n+ */\n+int ref_storage_backend_config(const char *var, const char *value, void *ptr);\n+\n+struct ref_storage_be;\n+/*\n * Switch to an alternate ref storage backend.\n */\n int set_ref_storage_backend(const char *name);\n \n int ref_storage_backend_exists(const char *name);\n \n+void register_ref_storage_backend(struct ref_storage_be *be);\n+\n #endif /* REFS_H */\ndiff --git a/setup.c b/setup.c\nindex 0deb022..1a62277 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"refs.h\"\n #include \"string-list.h\"\n \n static int inside_git_dir = -1;\n@@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n \treturn ret;\n }\n \n+int ref_storage_backend_config(const char *var, const char *value, void *ptr)\n+{\n+\tchar **cdata = ptr;\n+\n+\tif (!strcmp(var, \"extensions.refstorage\"))\n+\t\t*cdata = xstrdup(value);\n+\treturn 0;\n+}\n+\n /*\n * Test if it looks like we're at a git directory.\n * We want to see:\n@@ -390,6 +400,8 @@ static int check_repo_format(const char *var, const char *value, void *cb)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\trepository_format_precious_objects = git_config_bool(var, value);\n+\t\telse if (!strcmp(ext, \"refstorage\"))\n+\t\t\tref_storage_backend = xstrdup(value);\n \t\telse\n \t\t\tstring_list_append(&unknown_extensions, ext);\n \t}\ndiff --git a/t/t0001-init.sh b/t/t0001-init.sh\nindex 295aa59..31c8279 100755\n--- a/t/t0001-init.sh\n+++ b/t/t0001-init.sh\n@@ -174,6 +174,31 @@ test_expect_success 'reinit' '\n \ttest_i18ncmp again/empty again/err2\n '\n \n+test_expect_success 'init with bogus storage backend fails' '\n+\n+\t(\n+\t\tmkdir again2 &&\n+\t\tcd again2 &&\n+\t\ttest_must_fail git init --ref-storage=test >out2 2>err2 &&\n+\t\ttest_i18ngrep \"Unknown ref storage backend test\" err2\n+\t)\n+'\n+\n+test_expect_failure 'reinit with changed storage backend fails' '\n+\n+\t(\n+\t\tmkdir again3 &&\n+\t\tcd again3 &&\n+\t\tgit init >out1 2>err1 &&\n+\t\tgit init --ref-storage=test >out2 2>err2\n+\t) &&\n+\ttest_i18ngrep \"Initialized empty\" again3/out1 &&\n+\ttest_i18ngrep \"Reinitialized existing\" again3/out2 &&\n+\t>again3/empty &&\n+\ttest_i18ncmp again3/empty again3/err1 &&\n+\ttest_i18ncmp again3/empty again3/err2\n+'\n+\n test_expect_success 'init with --template' '\n \tmkdir template-source &&\n \techo content >template-source/file &&\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277579","messageId":"1454701462-3817-17-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 16/21] refs: check submodules ref storage config","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:17Z","receivedAt":"2016-02-05T19:44:17Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"All submodules must have the same ref storage (for now). Confirm that\nthis is so before attempting to do anything with submodule refs.\n\nSigned-off-by: David Turner \n---\n refs.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 8 ++------\n 2 files changed, 58 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 191b0a2..715a492 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -308,6 +308,54 @@ int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n \treturn for_each_ref_in(\"refs/tags/\", fn, cb_data);\n }\n \n+static int submodule_backend(const char *key, const char *value, void *data)\n+{\n+\tconst char **path = data;\n+\tchar **old_path = data;\n+\tif (!strcmp(key, \"extensions.refstorage\") &&\n+\t !git_config_string(path, key, \"extensions.refstorage\"))\n+\t\t\tfree(*old_path);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Check that a submodule exists. If its ref storage backend differs\n+ * from the current backend, die. If the submodule exists, return\n+ * 0. Else return -1.\n+ */\n+static int check_submodule_backend(const char *submodule)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tchar *submodule_storage_backend;\n+\tint result = -1;\n+\n+\tif (!submodule)\n+\t\treturn 0;\n+\n+\tsubmodule_storage_backend = xstrdup(\"files\");\n+\n+\tstrbuf_addstr(&sb, submodule);\n+\tif (!is_nonbare_repository_dir(&sb))\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_git_path_submodule(&sb, submodule, \"config\");\n+\n+\tgit_config_from_file(submodule_backend, sb.buf,\n+\t\t\t &submodule_storage_backend);\n+\tif (strcmp(submodule_storage_backend, ref_storage_backend))\n+\t\tdie(_(\"Ref storage '%s' for submodule '%s' does not match our storage, '%s'\"),\n+\t\t submodule_storage_backend, submodule, ref_storage_backend);\n+\n+\tresult = 0;\n+done:\n+\tfree(submodule_storage_backend);\n+\tstrbuf_release(&sb);\n+\n+\treturn result;\n+}\n+\n int for_each_tag_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n \treturn for_each_ref_in_submodule(submodule, \"refs/tags/\", fn, cb_data);\n@@ -320,6 +368,7 @@ int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n \n int for_each_branch_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n+\tcheck_submodule_backend(submodule);\n \treturn for_each_ref_in_submodule(submodule, \"refs/heads/\", fn, cb_data);\n }\n \n@@ -330,6 +379,7 @@ int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n \n int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n+\tcheck_submodule_backend(submodule);\n \treturn for_each_ref_in_submodule(submodule, \"refs/remotes/\", fn, cb_data);\n }\n \n@@ -1377,6 +1427,9 @@ int create_symref(const char *ref_target, const char *refs_heads_master,\n int resolve_gitlink_ref(const char *path, const char *refname,\n \t\t\tunsigned char *sha1)\n {\n+\tif (check_submodule_backend(path))\n+\t\treturn -1;\n+\n \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n }\n \n@@ -1387,6 +1440,7 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n+\tcheck_submodule_backend(submodule);\n \treturn the_refs_backend->head_ref_submodule(submodule, fn, cb_data);\n }\n \n@@ -1397,6 +1451,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n \n int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n+\tcheck_submodule_backend(submodule);\n \treturn the_refs_backend->for_each_ref_submodule(submodule, fn, cb_data);\n }\n \n@@ -1415,6 +1470,7 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data,\n int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n+\tcheck_submodule_backend(submodule);\n \treturn the_refs_backend->for_each_ref_in_submodule(submodule, prefix,\n \t\t\t\t\t\t\t fn, cb_data);\n }\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d4f9040..00cd1be 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1357,13 +1357,9 @@ static int files_resolve_gitlink_ref(const char *path, const char *refname,\n \n \tstrbuf_add(&submodule, path, len);\n \trefs = lookup_ref_cache(submodule.buf);\n-\tif (!refs) {\n-\t\tif (!is_nonbare_repository_dir(&submodule)) {\n-\t\t\tstrbuf_release(&submodule);\n-\t\t\treturn -1;\n-\t\t}\n+\tif (!refs)\n \t\trefs = create_ref_cache(submodule.buf);\n-\t}\n+\n \tstrbuf_release(&submodule);\n \n \tretval = resolve_gitlink_ref_recursive(refs, refname, sha1, 0);\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277580","messageId":"1454701462-3817-18-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 17/21] clone: allow ref storage backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:18Z","receivedAt":"2016-02-05T19:44:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a new option, --ref-storage, to allow the ref storage backend to\nbe set on new clones.\n\nSubmodules must use the same ref storage as the parent repository, so\nwe also pass the --ref-storage option option when cloning\nsubmodules.\n\nSigned-off-by: David Turner \nSigned-off-by: SZEDER Gábor \n---\n Documentation/git-clone.txt | 5 +++++\n builtin/clone.c | 5 +++++\n builtin/submodule--helper.c | 2 +-\n contrib/completion/git-completion.bash | 1 +\n git-submodule.sh | 13 +++++++++++++\n 5 files changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 45d74be..68f56a7 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t [-o ] [-b ] [-u ] [--reference ]\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n+\t [--ref-storage=]\n \t [--recursive | --recurse-submodules] [--jobs ] [--] \n \t []\n \n@@ -224,6 +225,10 @@ objects from the source repository into a pack in the cloned repository.\n \tThe number of submodules fetched at the same time.\n \tDefaults to the `submodule.fetchJobs` option.\n \n+--ref-storage=::\n+\tType of ref storage backend. Default is to use the original files\n+\tbased ref storage.\n+\n ::\n \tThe (possibly remote) repository to clone from. See the\n \t<> section below for more information on specifying\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 191f522..5ba0514 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -95,6 +95,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n \t\t\tN_(\"set config inside the new repository\")),\n+\tOPT_STRING(0, \"ref-storage\", &ref_storage_backend,\n+\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \tOPT_END()\n };\n \n@@ -733,6 +735,9 @@ static int checkout(void)\n \t\tif (max_jobs != -1)\n \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n \n+\t\targv_array_pushl(&args, \"--ref-storage\",\n+\t\t\t\t ref_storage_backend, NULL);\n+\n \t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n \t\targv_array_clear(&args);\n \t}\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 689c354..879d6fb 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -457,7 +457,7 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url\n \t\targv_array_pushl(&cp.args, \"--reference\", reference, NULL);\n \tif (gitdir && *gitdir)\n \t\targv_array_pushl(&cp.args, \"--separate-git-dir\", gitdir, NULL);\n-\n+\targv_array_pushl(&cp.args, \"--ref-storage\", ref_storage_backend, NULL);\n \targv_array_push(&cp.args, url);\n \targv_array_push(&cp.args, path);\n \ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 0138d03..cb9c473 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -1092,6 +1092,7 @@ _git_clone ()\n \t\t\t--depth\n \t\t\t--single-branch\n \t\t\t--branch\n+\t\t\t--ref-storage=\n \t\t\t\"\n \t\treturn\n \t\t;;\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 6fce0dc..e7ac98b 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -527,6 +527,14 @@ cmd_update()\n \t\t--checkout)\n \t\t\tupdate=\"checkout\"\n \t\t\t;;\n+\t\t--ref-storage=*)\n+\t\t\tref_storage=\"$1\"\n+\t\t\t;;\n+\t\t--ref-storage)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tref_storage=\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n \t\t--depth)\n \t\t\tcase \"$2\" in '') usage ;; esac\n \t\t\tdepth=\"--depth=$2\"\n@@ -560,6 +568,11 @@ cmd_update()\n \tif test -n \"$init\"\n \tthen\n \t\tcmd_init \"--\" \"$@\" || return\n+\telse\n+\t\tif test -n \"$ref_storage\"\n+\t\tthen\n+\t\t\tusage\n+\t\tfi\n \tfi\n \n \tgit submodule--helper update-clone ${GIT_QUIET:+--quiet} \\\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277578","messageId":"1454701462-3817-19-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 18/21] svn: learn ref-storage argument","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:19Z","receivedAt":"2016-02-05T19:44:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"git svn learns to pass the ref-storage command-line argument (to init\nand clone) through to git init.\n\nSigned-off-by: David Turner \nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-completion.bash | 2 +-\n git-svn.perl | 6 +++++-\n 2 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex cb9c473..ba4137d 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -2477,7 +2477,7 @@ _git_svn ()\n \t\t\t--branches= --stdlayout --minimize-url\n \t\t\t--no-metadata --use-svm-props --use-svnsync-props\n \t\t\t--rewrite-root= --prefix= --use-log-author\n-\t\t\t--add-author-from $remote_opts\n+\t\t\t--add-author-from --ref-storage= $remote_opts\n \t\t\t\"\n \t\tlocal cmt_opts=\"\n \t\t\t--edit --rmdir --find-copies-harder --copy-similarity=\ndiff --git a/git-svn.perl b/git-svn.perl\nindex fa5f253..15d1544 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -141,7 +141,7 @@ my %fc_opts = ( 'follow-parent|follow!' => \\$Git::SVN::_follow_parent,\n \t\t'localtime' => \\$Git::SVN::_localtime,\n \t\t%remote_opts );\n \n-my ($_trunk, @_tags, @_branches, $_stdlayout);\n+my ($_trunk, @_tags, @_branches, $_stdlayout, $_ref_storage);\n my %icv;\n my %init_opts = ( 'template=s' => \\$_template, 'shared:s' => \\$_shared,\n 'trunk|T=s' => \\$_trunk, 'tags|t=s@' => \\@_tags,\n@@ -153,6 +153,7 @@ my %init_opts = ( 'template=s' => \\$_template, 'shared:s' => \\$_shared,\n \t\t 'use-svnsync-props' => sub { $icv{useSvnsyncProps} = 1 },\n \t\t 'rewrite-root=s' => sub { $icv{rewriteRoot} = $_[1] },\n \t\t 'rewrite-uuid=s' => sub { $icv{rewriteUUID} = $_[1] },\n+\t\t 'ref-storage=s' => \\$_ref_storage,\n %remote_opts );\n my %cmt_opts = ( 'edit|e' => \\$_edit,\n \t\t'rmdir' => \\$Git::SVN::Editor::_rmdir,\n@@ -469,6 +470,9 @@ sub do_git_init_db {\n \t\t\t\tpush @init_db, \"--shared\";\n \t\t\t}\n \t\t}\n+\t\tif (defined $_ref_storage) {\n+\t\t push @init_db, \"--ref-storage=\" . $_ref_storage;\n+\t\t}\n \t\tcommand_noisy(@init_db);\n \t\t$_repository = Git->repository(Repository => \".git\");\n \t}\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277577","messageId":"1454701462-3817-20-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 19/21] refs: add register_ref_storage_backends()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:20Z","receivedAt":"2016-02-05T19:44:20Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This new function will register all known ref storage backends... once\nthere are any other than the default. For now, it's a no-op.\n\nSigned-off-by: David Turner \n---\n builtin/init-db.c | 3 +++\n config.c | 25 +++++++++++++++++++++++++\n refs.c | 8 ++++++++\n refs.h | 2 ++\n 4 files changed, 38 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex d331ce8..4209b67 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -226,6 +226,7 @@ static int create_default_files(const char *template_path)\n \tif (strcmp(ref_storage_backend, \"files\")) {\n \t\tgit_config_set(\"extensions.refStorage\", ref_storage_backend);\n \t\tgit_config_set(\"core.repositoryformatversion\", ref_storage_backend);\n+\t\tregister_ref_storage_backends();\n \t\tif (set_ref_storage_backend(ref_storage_backend))\n \t\t\tdie(_(\"Unknown ref storage backend %s\"),\n \t\t\t ref_storage_backend);\n@@ -503,6 +504,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix, init_db_options, init_db_usage, 0);\n \n+\tregister_ref_storage_backends();\n+\n \tif (requested_ref_storage_backend &&\n \t !ref_storage_backend_exists(requested_ref_storage_backend))\n \t\tdie(_(\"Unknown ref storage backend %s\"),\ndiff --git a/config.c b/config.c\nindex b95ac3a..b9ef223 100644\n--- a/config.c\n+++ b/config.c\n@@ -11,6 +11,7 @@\n #include \"strbuf.h\"\n #include \"quote.h\"\n #include \"hashmap.h\"\n+#include \"refs.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n \n@@ -1207,6 +1208,30 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n \t}\n \n \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n+\t\tchar *storage = NULL;\n+\n+\t\t/*\n+\t\t * make sure we always read the ref storage config\n+\t\t * from the extensions section on startup\n+\t\t */\n+\t\tret += git_config_from_file(ref_storage_backend_config,\n+\t\t\t\t\t repo_config, &storage);\n+\n+\t\tregister_ref_storage_backends();\n+\t\tif (!storage)\n+\t\t\tstorage = xstrdup(\"\");\n+\n+\t\tif ((!*storage) ||\n+\t\t (!strcmp(storage, \"files\"))) {\n+\t\t\t/* default backend, nothing to do */\n+\t\t\tfree(storage);\n+\t\t} else {\n+\t\t\tref_storage_backend = storage;\n+\t\t\tif (set_ref_storage_backend(ref_storage_backend))\n+\t\t\t\tdie(_(\"Unknown ref storage backend %s\"),\n+\t\t\t\t ref_storage_backend);\n+\t\t}\n+\n \t\tret += git_config_from_file(fn, repo_config, data);\n \t\tfound += 1;\n \t}\ndiff --git a/refs.c b/refs.c\nindex 715a492..e50cca0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1554,3 +1554,11 @@ done:\n \tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\n }\n+\n+void register_ref_storage_backends(void) {\n+\t/*\n+\t * No need to register the files backend; it's registered by\n+\t * default. Add register_ref_storage_backend(ptr-to-backend)\n+\t * entries below when you add a new backend.\n+\t */\n+}\ndiff --git a/refs.h b/refs.h\nindex 680a535..81aab6b 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -525,4 +525,6 @@ int ref_storage_backend_exists(const char *name);\n \n void register_ref_storage_backend(struct ref_storage_be *be);\n \n+void register_ref_storage_backends(void);\n+\n #endif /* REFS_H */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277581","messageId":"1454701462-3817-21-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:21Z","receivedAt":"2016-02-05T19:44:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a database backend for refs using LMDB. This backend runs git\nfor-each-ref about 30% faster than the files backend with fully-packed\nrefs on a repo with ~120k refs. It's also about 4x faster than using\nfully-unpacked refs. In addition, and perhaps more importantly, it\navoids case-conflict issues on OS X.\n\nLMDB has a few features that make it suitable for usage in git:\n\n1. It is relatively lightweight; it requires only one header file, and\nthe library code takes under 64k at runtime.\n\n2. It is well-tested: it's been used in OpenLDAP for years.\n\n3. It's very fast. LMDB's benchmarks show that it is among\nthe fastest key-value stores.\n\n4. It has a relatively simple concurrency story; readers don't\nblock writers and writers don't block readers.\n\nRonnie Sahlberg's original version of this patchset used tdb. The\nmajor disadvantage of tdb is that tdb is hard to build on OS X. It's\nalso not in homebrew. So lmdb seemed simpler.\n\nTo test this backend's correctness, I hacked test-lib.sh and\ntest-lib-functions.sh to run all tests under the refs backend. Dozens\nof tests use manual ref/reflog reading/writing, or create submodules\nwithout passing --ref-storage to git init. If those tests are\nchanged to use the update-ref machinery or test-refs-lmdb-backend (or,\nin the case of packed-refs, corrupt refs, and dumb fetch tests, are\nskipped), the only remaining failing tests are the git-new-workdir\ntests and the gitweb tests.\n\nSigned-off-by: David Turner \n---\n .gitignore | 1 +\n Documentation/config.txt | 7 +\n Documentation/git-clone.txt | 3 +-\n Documentation/git-init.txt | 2 +-\n Documentation/technical/refs-lmdb-backend.txt | 52 +\n Documentation/technical/repository-version.txt | 5 +\n Makefile | 12 +\n configure.ac | 33 +\n contrib/workdir/git-new-workdir | 3 +\n refs.c | 3 +\n refs.h | 2 +\n refs/lmdb-backend.c | 2052 ++++++++++++++++++++++++\n setup.c | 11 +-\n test-refs-lmdb-backend.c | 64 +\n transport.c | 7 +-\n 15 files changed, 2250 insertions(+), 7 deletions(-)\n create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n create mode 100644 refs/lmdb-backend.c\n create mode 100644 test-refs-lmdb-backend.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 1c2f832..87d45a2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -199,6 +199,7 @@\n /test-path-utils\n /test-prio-queue\n /test-read-cache\n+/test-refs-lmdb-backend\n /test-regex\n /test-revision-walking\n /test-run-command\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 06d3659..bc222c9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1153,6 +1153,13 @@ difftool..cmd::\n difftool.prompt::\n \tPrompt before each invocation of the diff tool.\n \n+extensions.refsStorage::\n+\tType of refs storage backend. Default is to use the original\n+\tfiles based ref storage. When set to \"lmdb\", refs are stored in\n+\tthe lmdb database backend. This setting reflects the refs\n+\tbackend that is currently in use; it is not possible to change\n+\tthe backend by changing this setting.\n+\n fetch.recurseSubmodules::\n \tThis option can be either set to a boolean value or to 'on-demand'.\n \tSetting it to a boolean changes the behavior of fetch and pull to\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 68f56a7..b9c52ce 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -227,7 +227,8 @@ objects from the source repository into a pack in the cloned repository.\n \n --ref-storage=::\n \tType of ref storage backend. Default is to use the original files\n-\tbased ref storage.\n+\tbased ref storage. Set to \"lmdb\" to store refs in the lmdb database\n+\tbackend.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex d2b150f..3b30bf3 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -116,7 +116,7 @@ does not exist, it will be created.\n --ref-storage=::\n Type of refs storage backend. Default is to use the original \"files\"\n storage, which stores ref data in files in .git/refs and\n-.git/packed-refs.\n+.git/packed-refs. Set to \"lmdb\" to activate the lmdb storage backend.\n \n TEMPLATE DIRECTORY\n ------------------\ndiff --git a/Documentation/technical/refs-lmdb-backend.txt b/Documentation/technical/refs-lmdb-backend.txt\nnew file mode 100644\nindex 0000000..d2cddb6\n--- /dev/null\n+++ b/Documentation/technical/refs-lmdb-backend.txt\n@@ -0,0 +1,52 @@\n+Notes on the LMDB refs backend\n+==============================\n+\n+Design:\n+------\n+\n+Refs and reflogs are stored in a lmdb database in .git/refdb. All\n+keys and values are \\0-terminated.\n+\n+Keys for refs are the name of the ref (e.g. refs/heads/master).\n+Values are the value of the ref, in hex\n+(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n+\n+All per-worktree refs (refs/bisect/* and HEAD) are stored using\n+the traditional files-based backend.\n+\n+Reflogs are stored as a series of database entries.\n+\n+For non-empty reflogs, there is one entry per logged ref update. The\n+key format is logs/[refname]\\0[timestamp]. The timestamp is a 64-bit\n+unsigned integer number of nanoseconds since 1/1/1970, stored in\n+network byte order. This means that reflog entries are\n+chronologically ordered. Because LMDB is a btree database, we can\n+efficiently iterate over these keys.\n+\n+For an empty reflog, there is a \"header\" entry to show that a reflog\n+exists. The header has the same format as an ordinary reflog, but with\n+a timestamp of all zeros and an empty value.\n+\n+Reflog values are in the same format as the original files-based\n+reflog, including the trailing LF. The date in the reflog value\n+matches the date in the timestamp field.\n+\n+Weaknesses:\n+-----------\n+\n+The reflog format is somewhat inefficient: a binary format could store\n+reflog date/time information in somewhat less space.\n+\n+The rsync and file:// transports don't work yet, because they\n+don't use the refs API.\n+\n+git new-workdir is incompatible with the lmdb backend. Fortunately,\n+git new-workdir is deprecated, and worktrees work fine.\n+\n+LMDB locks the entire database for write. Any other writer waits\n+until the first writer is done before beginning. Readers do not wait\n+for writers, and writers do not wait for readers. The underlying\n+scheme is approximately MVCC; each reader's queries see the state of\n+the database as-of the time that the reader acquired its read lock.\n+This is not too far off from the files backend, which loads all refs\n+into memory when one is requested.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 00ad379..fca5ecd 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -86,3 +86,8 @@ for testing format-1 compatibility.\n When the config key `extensions.preciousObjects` is set to `true`,\n objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n `git repack -d`).\n+\n+`refStorage`\n+~~~~~~~~~~~~\n+This extension allows the use of alternate ref storage backends. The\n+only defined value is `lmdb`.\ndiff --git a/Makefile b/Makefile\nindex 10566d6..fd80e94 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1042,6 +1042,17 @@ ifdef USE_LIBPCRE\n \tEXTLIBS += -lpcre\n endif\n \n+ifdef USE_LIBLMDB\n+\tBASIC_CFLAGS += -DUSE_LIBLMDB\n+\tifdef LIBLMDBDIR\n+\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n+\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n+\tendif\n+\tEXTLIBS += -llmdb\n+\tLIB_OBJS += refs/lmdb-backend.o\n+\tTEST_PROGRAMS_NEED_X += test-refs-lmdb-backend\n+endif\n+\n ifdef HAVE_ALLOCA_H\n \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n endif\n@@ -2140,6 +2151,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n+\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\ndiff --git a/configure.ac b/configure.ac\nindex 89e2590..3853bec 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -271,6 +271,24 @@ AS_HELP_STRING([], [ARG can be also prefix for libpcre library and hea\n dnl it yet.\n \tGIT_CONF_SUBST([LIBPCREDIR])\n fi)\n+\n+USE_LIBLMDB=YesPlease\n+AC_ARG_WITH(liblmdb,\n+AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])\n+AS_HELP_STRING([], [ARG can be also prefix for liblmdb library and headers]),\n+ if test \"$withval\" = \"no\"; then\n+\tUSE_LIBLMDB=\n+ elif test \"$withval\" = \"yes\"; then\n+\tUSE_LIBLMDB=YesPlease\n+ else\n+\tUSE_LIBLMDB=YesPlease\n+\tLIBLMDBDIR=$withval\n+\tAC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])\n+ dnl USE_LIBLMDB can still be modified below, so don't substitute\n+ dnl it yet.\n+\tGIT_CONF_SUBST([LIBLMDBDIR])\n+ fi)\n+\n #\n # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n AC_FUNC_ALLOCA\n@@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])\n \n fi\n \n+if test -n \"$USE_LIBLMDB\"; then\n+\n+GIT_STASH_FLAGS($LIBLMDBDIR)\n+\n+AC_CHECK_LIB([lmdb], [mdb_env_open],\n+[USE_LIBLMDB=YesPlease],\n+[USE_LIBLMDB=])\n+\n+GIT_UNSTASH_FLAGS($LIBLMDBDIR)\n+\n+GIT_CONF_SUBST([USE_LIBLMDB])\n+\n+fi\n+\n+\n #\n # Define NO_CURL if you do not have libcurl installed. git-http-pull and\n # git-http-push are not built, and you cannot use http:// and https://\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a..d153ddf 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -28,6 +28,9 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n git rev-parse --git-dir 2>/dev/null) ||\n die \"Not a git repository: \\\"$orig_git\\\"\"\n \n+ref_storage=$(git config extensions.refstorage || echo \"files\")\n+test \"$ref_storage\" != \"files\" && die \"git-new-workdir is incompatible with ref storage other than 'files'\"\n+\n case \"$git_dir\" in\n .git)\n \tgit_dir=\"$orig_git/.git\"\ndiff --git a/refs.c b/refs.c\nindex e50cca0..ba871d5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1561,4 +1561,7 @@ void register_ref_storage_backends(void) {\n \t * default. Add register_ref_storage_backend(ptr-to-backend)\n \t * entries below when you add a new backend.\n \t */\n+#ifdef USE_LIBLMDB\n+\tregister_ref_storage_backend(&refs_be_lmdb);\n+#endif\n }\ndiff --git a/refs.h b/refs.h\nindex 81aab6b..d57fe54 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -516,6 +516,8 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n int ref_storage_backend_config(const char *var, const char *value, void *ptr);\n \n struct ref_storage_be;\n+\n+extern struct ref_storage_be refs_be_lmdb;\n /*\n * Switch to an alternate ref storage backend.\n */\ndiff --git a/refs/lmdb-backend.c b/refs/lmdb-backend.c\nnew file mode 100644\nindex 0000000..138f391\n--- /dev/null\n+++ b/refs/lmdb-backend.c\n@@ -0,0 +1,2052 @@\n+/*\n+ * This file implements a lmdb backend for refs.\n+ *\n+ * The design of this backend relies on lmdb's write lock -- that is, any\n+ * write transaction blocks all other writers. Thus, as soon as a ref\n+ * transaction is opened, we know that any values we read won't\n+ * change out from under us, and we have a fully-consistent view of the\n+ * database.\n+ *\n+ * We store the content of refs including the trailing \\0 so that\n+ * standard C string functions can handle them. Just like struct\n+ * strbuf.\n+ */\n+#include \"../cache.h\"\n+#include \n+#include \"../object.h\"\n+#include \"../refs.h\"\n+#include \"refs-internal.h\"\n+#include \"../tag.h\"\n+#include \"../lockfile.h\"\n+\n+static MDB_env *env;\n+\n+static char *db_path;\n+\n+struct lmdb_transaction {\n+\tMDB_txn *txn;\n+\tMDB_dbi dbi;\n+\tMDB_cursor *cursor;\n+\tconst char *submodule;\n+\tint flags;\n+};\n+\n+struct lmdb_transaction transaction;\n+\n+static char *get_refdb_path(const char *base)\n+{\n+\tstatic struct strbuf path_buf = STRBUF_INIT;\n+\tstrbuf_reset(&path_buf);\n+\tstrbuf_addf(&path_buf, \"%s/refdb\", base);\n+\treturn path_buf.buf;\n+}\n+\n+static int in_write_transaction(void)\n+{\n+\treturn transaction.txn && !(transaction.flags & MDB_RDONLY);\n+}\n+\n+static void init_env(MDB_env **env, const char *path)\n+{\n+\tint ret;\n+\tif (*env)\n+\t\treturn;\n+\n+\tassert(path);\n+\n+\tret = mdb_env_create(env);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_env_create failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_set_maxreaders(*env, 1000);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_env_set_maxreaders failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_set_mapsize(*env, (1<<30));\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_set_mapsize failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_open(*env, path, 0 , 0664);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_env_open (%s) failed: %s\", path,\n+\t\t mdb_strerror(ret));\n+}\n+\n+static int lmdb_init_db(struct strbuf *err, int shared)\n+{\n+\t/*\n+\t * To create a db, all we need to do is make a directory for\n+\t * it to live in; lmdb will do the rest.\n+\t */\n+\n+\tif (!db_path)\n+\t\tdb_path = xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n+\n+\tif (mkdir(db_path, 0775) && errno != EEXIST) {\n+\t\tstrbuf_addf(err, \"%s\", strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void mdb_cursor_open_or_die(struct lmdb_transaction *transaction,\n+\t\t\t\t MDB_cursor **cursor)\n+{\n+\tint ret = mdb_cursor_open(transaction->txn, transaction->dbi, cursor);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_cursor_open failed: %s\", mdb_strerror(ret));\n+}\n+\n+static void submodule_path(struct strbuf *sb, const char *submodule,\n+\t\t\t const char *refname)\n+{\n+\tif (submodule)\n+\t\tstrbuf_git_path_submodule(sb, submodule, \"%s\", refname);\n+\telse\n+\t\tstrbuf_git_path(sb, \"%s\", refname);\n+}\n+\n+static int read_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t struct MDB_val *val, int *needs_free)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct stat st;\n+\tint ret = -1;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+#ifndef NO_SYMLINK_HEAD\n+\tif (lstat(path.buf, &st)) {\n+\t\tif (errno == ENOENT)\n+\t\t\tret = MDB_NOTFOUND;\n+\t\tgoto done;\n+\t}\n+\tif (S_ISLNK(st.st_mode)) {\n+\t\tstrbuf_readlink(&sb, path.buf, 0);\n+\t\tif (starts_with(sb.buf, \"refs/\") &&\n+\t\t !check_refname_format(sb.buf, 0)) {\n+\t\t\tval->mv_data = xstrfmt(\"ref: %s\", sb.buf);\n+\t\t\tval->mv_size = strlen(val->mv_data) + 1;\n+\t\t\tret = 0;\n+\t\t} else {\n+\t\t\tret = MDB_NOTFOUND;\n+\t\t}\n+\t\tstrbuf_release(&sb);\n+\t\tgoto done;\n+\t}\n+#endif\n+\n+\tif (strbuf_read_file(&sb, path.buf, 200) < 0) {\n+\t\tstrbuf_release(&sb);\n+\t\tif (errno == ENOENT)\n+\t\t\tret = MDB_NOTFOUND;\n+\t\tgoto done;\n+\t}\n+\tstrbuf_rtrim(&sb);\n+\n+\tval->mv_data = strbuf_detach(&sb, &val->mv_size);\n+\tval->mv_size++;\n+\n+\tret = 0;\n+done:\n+\tstrbuf_release(&path);\n+\t*needs_free = !ret;\n+\treturn ret;\n+}\n+\n+static void write_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t MDB_val *val)\n+{\n+\tstatic struct lock_file lock;\n+\tint fd;\n+\tint len = val->mv_size - 1;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\tsafe_create_leading_directories(path.buf);\n+\n+\tfd = hold_lock_file_for_update(&lock, path.buf, LOCK_DIE_ON_ERROR);\n+\tstrbuf_release(&path);\n+\n+\tif (write_in_full(fd, val->mv_data, len) != len ||\n+\t write_in_full(fd, \"\\n\", 1) != 1)\n+\t\tdie_errno(_(\"failed to write new HEAD\"));\n+\n+\tif (commit_lock_file(&lock))\n+\t\tdie_errno(_(\"failed to write new HEAD\"));\n+}\n+\n+static int del_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\tMDB_val *val)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint result;\n+\n+\t/*\n+\t * Returning deleted ref data is not yet implemented, but no\n+\t * callers need it.\n+\t */\n+\tassert(val == NULL);\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+\tresult = unlink(path.buf);\n+\tstrbuf_release(&path);\n+\tif (result && errno != ENOENT)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Read a ref. If the ref is a per-worktree ref, read it from disk.\n+ * Otherwise, read it from LMDB. LMDB manages its own memory, so the\n+ * data returned in *val will ordinarily not need to be freed. But\n+ * when a per-worktree ref is (successfully) read, non-LMDB memory is\n+ * allocated. In this case, *needs_free is set so that the caller can\n+ * free the memory when it is done with it.\n+ */\n+static int mdb_get_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val, int *needs_free)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\treturn read_per_worktree_ref(transaction->submodule,\n+\t\t\t\t\t key->mv_data, val, needs_free);\n+\n+\t*needs_free = 0;\n+\tret = mdb_get(transaction->txn, transaction->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"BUG: mdb_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static int mdb_del_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\tdie(\"BUG: this backend should only try to delete normal refs\");\n+\n+\tret = mdb_del(transaction->txn, transaction->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"BUG: mdb_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static void mdb_put_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\tdie(\"BUG: this backend should only try to write normal refs\");\n+\n+\tret = mdb_put(transaction->txn, transaction->dbi, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret == MDB_BAD_VALSIZE)\n+\t\t\tdie(_(\"Ref name %s too long (max size is %d)\"),\n+\t\t\t (const char *)key->mv_data,\n+\t\t\t mdb_env_get_maxkeysize(env));\n+\t\telse\n+\t\t\tdie(\"BUG: mdb_put failed: (%s -> %s) %s\",\n+\t\t\t (const char *)key->mv_data,\n+\t\t\t (const char *)val->mv_data, mdb_strerror(ret));\n+\t}\n+}\n+\n+static int mdb_cursor_get_or_die(MDB_cursor *cursor, MDB_val *key, MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tret = mdb_cursor_get(cursor, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"BUG: mdb_cursor_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\tassert(((char *)val->mv_data)[val->mv_size - 1] == 0);\n+\treturn 0;\n+}\n+\n+static int mdb_cursor_del_or_die(MDB_cursor *cursor, int flags)\n+{\n+\tint ret = mdb_cursor_del(cursor, flags);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"BUG: mdb_cursor_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * Begin a transaction. Because only one transaction per thread is\n+ * permitted, we use a global transaction object. If a read-write\n+ * transaction is presently already in-progress, and a read-only\n+ * transaction is requested, the read-write transaction will be\n+ * returned instead. If a read-write transaction is requested and a\n+ * read-only transaction is open, the read-only transaction will be\n+ * closed.\n+ *\n+ * It is a bug to request a read-write transaction during another\n+ * read-write transaction.\n+ *\n+ * As a result, it is unsafe to retain read-only transactions past the\n+ * point where a read-write transaction might be needed. For\n+ * instance, any call that has callbacks outside this module must\n+ * conclude all of its reads from the database before calling those\n+ * callbacks, or must reacquire the transaction after its callbacks\n+ * are completed.\n+ */\n+int lmdb_transaction_begin_flags(struct strbuf *err, unsigned int flags)\n+{\n+\tint ret;\n+\tMDB_txn *txn;\n+\tstatic size_t last_txnid = 0;\n+\tint force_restart = 0;\n+\tMDB_envinfo stat;\n+\n+\tif (!db_path)\n+\t\tdb_path = xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n+\tinit_env(&env, db_path);\n+\n+\t/*\n+\t * Since each transaction sees a consistent view of the db,\n+\t * downstream processes that write the db won't be seen in\n+\t * this transaction. We can check if the last transaction id\n+\t * has changed since this read transaction was started, and if\n+\t * so, we want to reopen the transaction.\n+\t */\n+\n+\tmdb_env_info(env, &stat);\n+\tif (stat.me_last_txnid != last_txnid) {\n+\t\tforce_restart = 1;\n+\t\tlast_txnid = stat.me_last_txnid;\n+\t}\n+\n+\tif (!transaction.txn) {\n+\t\tret = mdb_txn_begin(env, NULL, flags, &txn);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"BUG: mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"BUG: mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t\treturn 0;\n+\t}\n+\n+\tif (transaction.flags == flags && !(flags & MDB_RDONLY))\n+\t\tdie(\"BUG: rw transaction started during another rw txn\");\n+\n+\tif (force_restart || (transaction.flags != flags && transaction.flags & MDB_RDONLY)) {\n+\t\t/*\n+\t\t * RO -> RW, or forced restart due to possible changes\n+\t\t * from downstream processes.\n+\t\t */\n+\t\tmdb_txn_abort(transaction.txn);\n+\t\tret = mdb_txn_begin(env, NULL, flags, &txn);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"BUG: restarting txn: mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"BUG: mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t}\n+\t/* RW -> RO just keeps the RW txn */\n+\treturn 0;\n+}\n+\n+static struct lmdb_transaction *lmdb_transaction_begin_flags_or_die(int flags)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tif (lmdb_transaction_begin_flags(&err, flags))\n+\t\tdie(\"%s\", err.buf);\n+\treturn &transaction;\n+}\n+\n+#define MAXDEPTH 5\n+\n+static const char *parse_ref_data(struct lmdb_transaction *transaction,\n+\t\t\t\t const char *refname, const char *ref_data,\n+\t\t\t\t unsigned char *sha1, int resolve_flags,\n+\t\t\t\t int *flags, int bad_name)\n+{\n+\tint depth = MAXDEPTH;\n+\tconst char *buf;\n+\tstatic struct strbuf refname_buffer = STRBUF_INIT;\n+\tstatic struct strbuf refdata_buffer = STRBUF_INIT;\n+\tMDB_val key, val;\n+\tint needs_free = 0;\n+\n+\tfor (;;) {\n+\t\tif (--depth < 0)\n+\t\t\treturn NULL;\n+\n+\t\tif (!starts_with(ref_data, \"ref:\")) {\n+\t\t\tif (get_sha1_hex(ref_data, sha1) ||\n+\t\t\t (ref_data[40] != '\\0' && !isspace(ref_data[40]))) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tif (bad_name) {\n+\t\t\t\thashclr(sha1);\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t} else if (is_null_sha1(sha1)) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (flags)\n+\t\t\t*flags |= REF_ISSYMREF;\n+\t\tbuf = ref_data + 4;\n+\t\twhile (isspace(*buf))\n+\t\t\tbuf++;\n+\t\tstrbuf_reset(&refname_buffer);\n+\t\tstrbuf_addstr(&refname_buffer, buf);\n+\t\trefname = refname_buffer.buf;\n+\t\tif (resolve_flags & RESOLVE_REF_NO_RECURSE) {\n+\t\t\thashclr(sha1);\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\n+\t\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t\t !refname_is_safe(buf)) {\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\tbad_name = 1;\n+\t\t}\n+\n+\t\tkey.mv_data = (char *)refname;\n+\t\tkey.mv_size = strlen(refname) + 1;\n+\t\tif (mdb_get_or_die(transaction, &key, &val, &needs_free)) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (bad_name) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\t\treturn NULL;\n+\n+\t\t\treturn refname;\n+\t\t}\n+\t\tstrbuf_reset(&refdata_buffer);\n+\t\tstrbuf_add(&refdata_buffer, val.mv_data, val.mv_size);\n+\t\tif (needs_free)\n+\t\t\tfree(val.mv_data);\n+\t\tref_data = refdata_buffer.buf;\n+\t}\n+\treturn refname;\n+}\n+\n+static int verify_refname_available_txn(struct lmdb_transaction *transaction,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tstruct string_list *extras,\n+\t\t\t\t\tstruct string_list *skip,\n+\t\t\t\t\tstruct strbuf *err)\n+{\n+\tMDB_cursor *cursor;\n+\tMDB_val key;\n+\tMDB_val val;\n+\tint mdb_ret;\n+\tsize_t refname_len;\n+\tchar *search_key;\n+\tconst char *extra_refname;\n+\tint ret = 1;\n+\tsize_t i;\n+\n+\tmdb_cursor_open_or_die(transaction, &cursor);\n+\n+\trefname_len = strlen(refname) + 2;\n+\tkey.mv_size = refname_len;\n+\tsearch_key = xmalloc(refname_len);\n+\tmemcpy(search_key, refname, refname_len - 2);\n+\tsearch_key[refname_len - 2] = '/';\n+\tsearch_key[refname_len - 1] = 0;\n+\tkey.mv_data = search_key;\n+\n+\t/* Check for subdirs of refname: we start at refname/ */\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (starts_with(key.mv_data, refname) &&\n+\t\t ((char *)key.mv_data)[refname_len - 2] == '/') {\n+\t\t\tif (skip && string_list_has_string(skip, key.mv_data))\n+\t\t\t\tgoto next;\n+\n+\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"), (char *)key.mv_data, refname);\n+\t\t\tgoto done;\n+\t\t}\n+\t\tbreak;\n+\tnext:\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\t/* Check for parent dirs of refname. */\n+\tfor (i = 0; i < refname_len - 2; i++) {\n+\t\tif (search_key[i] == '/') {\n+\t\t\tsearch_key[i] = 0;\n+\t\t\tif (skip && string_list_has_string(skip, search_key)) {\n+\t\t\t\tsearch_key[i] = '/';\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, search_key)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, search_key);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tkey.mv_data = search_key;\n+\t\t\tkey.mv_size = i + 1;\n+\t\t\tif (!mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"), (char *)key.mv_data, refname);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tsearch_key[i] = '/';\n+\t\t}\n+\t}\n+\n+\textra_refname = find_descendant_ref(refname, extras, skip);\n+\tif (extra_refname) {\n+\t\tstrbuf_addf(err,\n+\t\t\t _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t refname, extra_refname);\n+\t\tret = 1;\n+\t} else {\n+\t\tret = 0;\n+\t}\n+done:\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\treturn ret;\n+}\n+\n+static const char *resolve_ref_unsafe_txn(struct lmdb_transaction *transaction,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n+{\n+\tint bad_name = 0;\n+\tchar *ref_data;\n+\tstruct MDB_val key, val;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint needs_free = 0;\n+\tconst char *ret;\n+\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\n+\tif (flags)\n+\t\t*flags = 0;\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (flags)\n+\t\t\t*flags |= REF_BAD_NAME;\n+\n+\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t !refname_is_safe(refname)) {\n+\t\t\terrno = EINVAL;\n+\t\t\treturn NULL;\n+\t\t}\n+\t\t/*\n+\t\t * dwim_ref() uses REF_ISBROKEN to distinguish between\n+\t\t * missing refs and refs that were present but invalid,\n+\t\t * to complain about the latter to stderr.\n+\t\t *\n+\t\t * We don't know whether the ref exists, so don't set\n+\t\t * REF_ISBROKEN yet.\n+\t\t */\n+\t\tbad_name = 1;\n+\t}\n+\n+\tkey.mv_data = (void *)refname;\n+\tkey.mv_size = strlen(refname) + 1;\n+\tif (mdb_get_or_die(transaction, &key, &val, &needs_free)) {\n+\t\tif (bad_name) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\treturn NULL;\n+\n+\t\tif (verify_refname_available_txn(transaction, refname, NULL, NULL, &err)) {\n+\t\t\terror(\"%s\", err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\thashclr(sha1);\n+\t\treturn refname;\n+\t}\n+\n+\tref_data = val.mv_data;\n+\tassert(ref_data[val.mv_size - 1] == 0);\n+\n+\tret = parse_ref_data(transaction, refname, ref_data, sha1,\n+\t\t\t resolve_flags, flags, bad_name);\n+\tif (needs_free)\n+\t\tfree(ref_data);\n+\treturn ret;\n+}\n+\n+static const char *lmdb_resolve_ref_unsafe(const char *refname, int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn resolve_ref_unsafe_txn(&transaction, refname,\n+\t\t\t\t resolve_flags, sha1, flags);\n+}\n+\n+static void write_u64(char *buf, uint64_t number)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < 8; i++)\n+\t\tbuf[i] = (number >> (i * 8)) & 0xff;\n+}\n+\n+static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tchar *email_end, *message;\n+\tunsigned long timestamp;\n+\tint tz;\n+\n+\t/* old (raw sha) new (raw sha) name SP time TAB msg LF */\n+\tif (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n+\t !(email_end = strchr(sb->buf + 40, '>')) ||\n+\t email_end[1] != ' ')\n+\t\treturn 0; /* corrupt? */\n+\n+\ttimestamp = strtoul(email_end + 2, &message, 10);\n+\n+\tif (!timestamp ||\n+\t !message || message[0] != ' ' ||\n+\t (message[1] != '+' && message[1] != '-') ||\n+\t !isdigit(message[2]) || !isdigit(message[3]) ||\n+\t !isdigit(message[4]) || !isdigit(message[5]))\n+\t\treturn 0; /* corrupt? */\n+\n+\thashcpy(osha1, (const unsigned char *)sb->buf);\n+\thashcpy(nsha1, (const unsigned char *)sb->buf + 20);\n+\n+\temail_end[1] = '\\0';\n+\ttz = strtol(message + 1, NULL, 10);\n+\tif (message[6] != '\\t')\n+\t\tmessage += 6;\n+\telse\n+\t\tmessage += 7;\n+\treturn fn(osha1, nsha1, sb->buf + 40, timestamp, tz, message, cb_data);\n+}\n+\n+static void format_reflog_entry(struct strbuf *buf,\n+\t\t\t\tconst unsigned char *old_sha1,\n+\t\t\t\tconst unsigned char *new_sha1,\n+\t\t\t\tconst char *committer, const char *msg)\n+{\n+\tint len;\n+\tint msglen;\n+\n+\tassert(buf->len == 0);\n+\tstrbuf_add(buf, old_sha1, 20);\n+\tstrbuf_add(buf, new_sha1, 20);\n+\tstrbuf_addstr(buf, committer);\n+\tstrbuf_addch(buf, '\\n');\n+\n+\tlen = buf->len;\n+\tmsglen = msg ? strlen(msg) : 0;\n+\tif (msglen) {\n+\t\tint copied;\n+\t\tstrbuf_grow(buf, msglen + 1);\n+\t\tcopied = copy_reflog_msg(buf->buf + 40 + strlen(committer), msg) - 1;\n+\t\tbuf->len = len + copied;\n+\t\tbuf->buf[buf->len] = 0;\n+\t}\n+}\n+\n+static int log_ref_write(const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const char *msg,\n+\t\t\t int flags,\n+\t\t\t struct strbuf *err)\n+{\n+\tMDB_val key, val;\n+\tuint64_t now = getnanotime();\n+\tint result;\n+\tstruct strbuf log_key = STRBUF_INIT;\n+\tint refname_len;\n+\tMDB_cursor *cursor;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *timestamp;\n+\tuint64_t zero = 0;\n+\n+\tif (log_all_ref_updates < 0)\n+\t\tlog_all_ref_updates = !is_bare_repository();\n+\n+\t/* it is assumed that we are in a ref transaction here */\n+\tassert(transaction.txn);\n+\n+\tresult = safe_create_reflog(refname, flags & REF_FORCE_CREATE_REFLOG, err);\n+\tif (result)\n+\t\treturn result;\n+\n+\t/* \"logs/\" + refname + \\0 + 8-byte timestamp for sorting and expiry. */\n+\trefname_len = strlen(refname);\n+\n+\tstrbuf_addf(&log_key, \"logs/%s\", refname);\n+\tstrbuf_add(&log_key, &zero, 8);\n+\tkey.mv_data = log_key.buf;\n+\tkey.mv_size = log_key.len + 1;\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\t/* if no reflog exists, we're done */\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||\n+\t strcmp(key.mv_data, log_key.buf))\n+\t\tgoto done;\n+\n+\t/* Is this a header? We only need the header for empty reflogs */\n+\ttimestamp = (const char *)key.mv_data + refname_len + 6;\n+\tif (ntohll(*(uint64_t *)timestamp) == 0)\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\n+\tkey.mv_data = log_key.buf;\n+\n+\twrite_u64((char *)key.mv_data + refname_len + 6, htonll(now));\n+\n+\tformat_reflog_entry(&buf, old_sha1, new_sha1,\n+\t\t\t git_committer_info(0), msg);\n+\tassert(buf.len >= 42);\n+\tval.mv_data = buf.buf;\n+\tval.mv_size = buf.len + 1;\n+\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\tstrbuf_release(&buf);\n+\n+done:\n+\tstrbuf_release(&log_key);\n+\tmdb_cursor_close(cursor);\n+\treturn 0;\n+}\n+\n+static int lmdb_verify_refname_available(const char *refname,\n+\t\t\t\t\t struct string_list *extras,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn verify_refname_available_txn(&transaction, refname, extras, skip, err);\n+}\n+\n+/*\n+ * Attempt to resolve `refname` to `old_sha1` (if old_sha1 is\n+ * non-null). The return value is a pointer to a newly-allocated\n+ * string containing the next ref name that this resolves to. So if\n+ * HEAD is a symbolic ref to refs/heads/example, which is itself a\n+ * symbolic ref to refs/heads/foo, return refs/heads/example,\n+ * and fill in resolved_sha1 with the sha of refs/heads/foo.\n+ */\n+static char *check_ref(MDB_txn *txn, const char *refname,\n+\t\t const unsigned char *old_sha1,\n+\t\t unsigned char *resolved_sha1, int flags,\n+\t\t int *type_p)\n+{\n+\tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n+\tint resolve_flags = 0;\n+\tint type;\n+\tchar *resolved_refname;\n+\n+\tif (mustexist)\n+\t\tresolve_flags |= RESOLVE_REF_READING;\n+\tif (flags & REF_DELETING) {\n+\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n+\t\tif (flags & REF_NODEREF)\n+\t\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n+\t}\n+\n+\t/*\n+\t * The first time we resolve the refname, we're just trying to\n+\t * see if there is any ref at all by this name, even if it is\n+\t * a broken symref.\n+\t */\n+\trefname = resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t resolved_sha1, &type);\n+\tif (type_p)\n+\t\t*type_p = type;\n+\n+\tif (!refname)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * Need to copy refname here because the resolve_ref_unsafe\n+\t * returns a pointer to a static buffer that could get mangled\n+\t * by the second call.\n+\t */\n+\tresolved_refname = xstrdup(refname);\n+\n+\tif (old_sha1) {\n+\t\tif (flags & REF_NODEREF) {\n+\t\t\tresolve_flags &= ~RESOLVE_REF_NO_RECURSE;\n+\n+\t\t\tresolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t resolved_sha1, &type);\n+\t\t}\n+\t\tif (hashcmp(old_sha1, resolved_sha1)) {\n+\t\t\terror(_(\"ref %s is at %s but expected %s\"), refname,\n+\t\t\t sha1_to_hex(resolved_sha1), sha1_to_hex(old_sha1));\n+\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\treturn resolved_refname;\n+}\n+\n+static int mdb_transaction_commit(struct lmdb_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint result;\n+\n+\tresult = mdb_txn_commit(transaction->txn);\n+\tif (result && err)\n+\t\tstrbuf_addstr(err, mdb_strerror(result));\n+\n+\ttransaction->txn = NULL;\n+\treturn result;\n+}\n+\n+static int lmdb_delete_reflog(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tint mdb_ret;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint in_transaction;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.delete_reflog(refname);\n+\n+\tin_transaction = in_write_transaction();\n+\n+\tlog_path = xstrfmt(\"logs/%s\", refname);\n+\tlen = strlen(log_path) + 1;\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) ||\n+\t\t ((char *)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\ttransaction.cursor = NULL;\n+\n+\tif (!in_transaction && mdb_transaction_commit(&transaction, &err)) {\n+\t\twarning(\"%s\", err.buf);\n+\t\tret = 01;\n+\t}\n+\tstrbuf_release(&err);\n+\treturn ret;\n+}\n+\n+#define REF_NO_REFLOG 0x8000\n+\n+static int lmdb_transaction_update(const char *refname,\n+\t\t\t\t const unsigned char *new_sha1,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tconst char *orig_refname = refname;\n+\tMDB_val key, val;\n+\tunsigned char resolved_sha1[20];\n+\tint type;\n+\tint ret = -1;\n+\n+\tif ((flags & REF_HAVE_NEW) && is_null_sha1(new_sha1))\n+\t\tflags |= REF_DELETING;\n+\n+\tif (new_sha1 && !is_null_sha1(new_sha1) &&\n+\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(err, _(\"refusing to update ref with bad name %s\"),\n+\t\t\t refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\trefname = check_ref(transaction.txn, orig_refname, old_sha1,\n+\t\t\t resolved_sha1, flags, &type);\n+\tif (refname == NULL) {\n+\t\tstrbuf_addf(err, _(\"cannot lock the ref '%s'\"), orig_refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\tif (!(flags & REF_DELETING) && is_null_sha1(resolved_sha1) &&\n+\t verify_refname_available_txn(&transaction, refname, NULL, NULL, err))\n+\t\treturn TRANSACTION_NAME_CONFLICT;\n+\n+\tif (flags & REF_NODEREF) {\n+\t\tfree((void *)refname);\n+\t\trefname = orig_refname;\n+\t}\n+\n+\tkey.mv_size = strlen(refname) + 1;\n+\tkey.mv_data = (void *)refname;\n+\n+\tif ((flags & REF_HAVE_NEW) && !is_null_sha1(new_sha1)) {\n+\t\tint overwriting_symref = ((type & REF_ISSYMREF) &&\n+\t\t\t\t\t (flags & REF_NODEREF));\n+\n+\t\tstruct object *o = parse_object(new_sha1);\n+\t\tif (!o) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Trying to write ref %s with nonexistent object %s\"),\n+\t\t\t\t refname, sha1_to_hex(new_sha1));\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (o->type != OBJ_COMMIT && is_branch(refname)) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Trying to write non-commit object %s to branch %s\"),\n+\t\t\t\t sha1_to_hex(new_sha1), refname);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (!overwriting_symref\n+\t\t && !hashcmp(resolved_sha1, new_sha1)) {\n+\t\t\t/*\n+\t\t\t * The reference already has the desired\n+\t\t\t * value, so we don't need to write it.\n+\t\t\t */\n+\t\t\tflags |= REF_NO_REFLOG;\n+\t\t} else {\n+\t\t\tval.mv_size = 41;\n+\t\t\tif (new_sha1)\n+\t\t\t\tval.mv_data = sha1_to_hex(new_sha1);\n+\t\t\telse\n+\t\t\t\tval.mv_data = sha1_to_hex(null_sha1);\n+\t\t\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\t\t}\n+\t}\n+\n+\tif (flags & REF_DELETING) {\n+\t\tif (mdb_del_or_die(&transaction, &key, NULL)) {\n+\t\t\tif (old_sha1 && !is_null_sha1(old_sha1)) {\n+\t\t\t\tstrbuf_addf(err, _(\"No such ref %s\"), refname);\n+\t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t\tlmdb_delete_reflog(orig_refname);\n+\t} else if (!(flags & REF_NO_REFLOG)) {\n+\t\tif (!new_sha1)\n+\t\t\tnew_sha1 = null_sha1;\n+\t\tif (log_ref_write(orig_refname, resolved_sha1,\n+\t\t\t\t new_sha1, msg, flags, err) < 0)\n+\t\t\tgoto done;\n+\t\tif (strcmp(refname, orig_refname) &&\n+\t\t log_ref_write(refname, resolved_sha1,\n+\t\t\t\t new_sha1, msg, flags, err) < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = 0;\n+done:\n+\tif (refname != orig_refname)\n+\t\tfree((void *) refname);\n+\treturn ret;\n+}\n+\n+static int lmdb_transaction_commit(struct ref_transaction *ref_transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint ret = 0, i;\n+\tint n = ref_transaction->nr;\n+\tstruct ref_update **updates = ref_transaction->updates;\n+\n+\t/*\n+\t * We might already be in a write transaction, because some\n+\t * lmdb backend functionality is implemented in terms of\n+\t * (other stuff) + ref_transaction_commit\n+\t */\n+\tif (!in_write_transaction())\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tstruct ref_update *update = updates[i];\n+\n+\t\tif (lmdb_transaction_update(update->refname,\n+\t\t\t\t\t update->new_sha1,\n+\t\t\t\t\t (update->flags & REF_HAVE_OLD) ?\n+\t\t\t\t\t update->old_sha1 : NULL,\n+\t\t\t\t\t update->flags,\n+\t\t\t\t\t update->msg,\n+\t\t\t\t\t err)) {\n+\t\t\tmdb_txn_abort(transaction.txn);\n+\t\t\tret = -1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t}\n+\tret = mdb_transaction_commit(&transaction, err);\n+\n+cleanup:\n+\tref_transaction->state = REF_TRANSACTION_CLOSED;\n+\treturn ret;\n+}\n+\n+static int rename_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\n+\tconst char *newrefname = cb_data;\n+\tMDB_val key, new_key, val;\n+\tstruct strbuf new_key_buf = STRBUF_INIT;\n+\n+\tassert(transaction.cursor);\n+\n+\tif (mdb_cursor_get_or_die(transaction.cursor, &key, &val, MDB_GET_CURRENT))\n+\t\tdie(\"BUG: renaming ref: mdb_cursor_get failed to get current\");\n+\n+\t/* This must really be a reflog entry */\n+\tassert(val.mv_size > 42);\n+\n+\tstrbuf_addf(&new_key_buf, \"logs/%s\", newrefname);\n+\tstrbuf_add(&new_key_buf, key.mv_data + key.mv_size - 8, 8);\n+\n+\tnew_key.mv_size = new_key_buf.len + 1;\n+\tnew_key.mv_data = new_key_buf.buf;\n+\tmdb_put_or_die(&transaction, &new_key, &val, 0);\n+\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\tstrbuf_release(&new_key_buf);\n+\treturn 0;\n+}\n+\n+static int lmdb_rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\tunsigned char orig_sha1[20];\n+\tint flag = 0;\n+\tint log = reflog_exists(oldref);\n+\tconst char *symref = NULL;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *ref_transaction;\n+\n+\tif (!strcmp(oldref, newref))\n+\t\treturn 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tref_transaction = ref_transaction_begin(&err);\n+\tif (!ref_transaction)\n+\t\tdie(\"%s\", err.buf);\n+\n+\tsymref = resolve_ref_unsafe(oldref, RESOLVE_REF_READING,\n+\t\t\t\t orig_sha1, &flag);\n+\tif (flag & REF_ISSYMREF) {\n+\t\terror(_(\"refname %s is a symbolic ref, renaming it is not supported\"),\n+\t\t oldref);\n+\t\tgoto fail;\n+\t}\n+\tif (!symref) {\n+\t\tmdb_txn_abort(transaction.txn);\n+\t\terror(_(\"refname %s not found\"), oldref);\n+\t\tgoto fail;\n+\t}\n+\tif (!rename_ref_available(oldref, newref))\n+\t\tgoto fail;\n+\n+\t/* Copy the reflog from the old to the new */\n+\tif (log) {\n+\t\tstruct strbuf old_log_sentinel = STRBUF_INIT;\n+\t\tMDB_val key;\n+\t\tint log_all;\n+\n+\t\tlog_all = log_all_ref_updates;\n+\t\tlog_all_ref_updates = 1;\n+\t\tif (safe_create_reflog(newref, 0, &err)) {\n+\t\t\terror(_(\"can't create reflog for %s: %s\"), newref, err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\tgoto fail;\n+\t\t}\n+\t\tlog_all_ref_updates = log_all;\n+\n+\t\tfor_each_reflog_ent(oldref, rename_reflog_ent, (void *)newref);\n+\t\tstrbuf_addf(&old_log_sentinel, \"logs/%sxxxxxxxx\", oldref);\n+\t\tmemset(old_log_sentinel.buf + old_log_sentinel.len - 8, 0, 8);\n+\n+\t\tkey.mv_size = old_log_sentinel.len;\n+\t\tkey.mv_data = old_log_sentinel.buf;\n+\n+\t\t/* It's OK if the old reflog is missing */\n+\t\tmdb_del_or_die(&transaction, &key, NULL);\n+\t\tstrbuf_release(&old_log_sentinel);\n+\t}\n+\n+\tif (ref_transaction_delete(ref_transaction, oldref,\n+\t\t\t\t orig_sha1, REF_NODEREF, NULL, &err)) {\n+\t\terror(_(\"unable to delete old %s\"), oldref);\n+\t\tgoto fail;\n+\t}\n+\n+\tif (ref_transaction_update(ref_transaction, newref, orig_sha1, NULL,\n+\t\t\t\t 0, logmsg, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tgoto fail;\n+\t}\n+\n+\tif (ref_transaction_commit(ref_transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tgoto fail;\n+\t}\n+\n+\treturn 0;\n+\n+fail:\n+\tref_transaction_free(ref_transaction);\n+\tstrbuf_release(&err);\n+\tmdb_txn_abort(transaction.txn);\n+\treturn 1;\n+}\n+\n+static int lmdb_delete_refs(struct string_list *refnames)\n+{\n+\tint i;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint result = 0;\n+\n+\tif (!refnames->nr)\n+\t\treturn 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tfor (i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\n+\t\tif (lmdb_transaction_update(refname, null_sha1, NULL,\n+\t\t\t\t\t REF_DELETING, NULL, &err))\n+\t\t\tresult |= error(_(\"could not remove reference %s: %s\"),\n+\t\t\t\t\trefname, err.buf);\n+\t}\n+\n+\tresult |= mdb_transaction_commit(&transaction, &err);\n+\tstrbuf_release(&err);\n+\treturn 0;\n+}\n+\n+static int lmdb_for_each_reflog_ent_order(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data, int reverse)\n+{\n+\tMDB_val key, val;\n+\tchar *search_key;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum MDB_cursor_op direction = reverse ? MDB_PREV : MDB_NEXT;\n+\tuint64_t zero = 0ULL;\n+\n+\tlog_path = xstrfmt(\"logs/%s\", refname);\n+\tlen = strlen(log_path) + 1;\n+\n+\tif (reverse) {\n+\t\t/*\n+\t\t * For a reverse search, start at the key\n+\t\t * lexicographically after the searched-for key.\n+\t\t * That's the one with \\1 appended to the key.\n+\t\t */\n+\t\tsearch_key = xstrfmt(\"%s\\1\", log_path);\n+\t\tkey.mv_size = len + 1;\n+\t} else {\n+\t\tsearch_key = xstrdup(log_path);\n+\t\tkey.mv_size = len;\n+\t}\n+\n+\tkey.mv_data = search_key;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\ttransaction.cursor = cursor;\n+\n+\t/*\n+\t * MDB's cursor API requires that the first mdb_cursor_get be\n+\t * called with MDB_SET_RANGE. For reverse searches, this will\n+\t * give us the entry one-past the entry we're looking for, so\n+\t * we should jump back using MDB_PREV.\n+\t */\n+\tmdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\tif (direction == MDB_PREV)\n+\t\tmdb_cursor_get_or_die(cursor, &key, &val, direction);\n+\n+\tdo {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) || ((char *)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tif (!memcmp(&zero, ((char *)key.mv_data) + key.mv_size - 8, 8))\n+\t\t\tcontinue;\n+\n+\t\tassert(val.mv_size != 0);\n+\n+\t\tstrbuf_add(&sb, val.mv_data, val.mv_size - 1);\n+\t\tret = show_one_reflog_ent(&sb, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\n+\t\tstrbuf_reset(&sb);\n+\t} while (!mdb_cursor_get_or_die(cursor, &key, &val, direction));\n+\n+\tstrbuf_release(&sb);\n+\tfree(log_path);\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.for_each_reflog_ent(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 0);\n+}\n+\n+static int lmdb_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.for_each_reflog_ent_reverse(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 1);\n+}\n+\n+static int lmdb_reflog_exists(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 1;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.reflog_exists(refname);\n+\n+\tlog_path = xstrfmt(\"logs/%s\", refname);\n+\tlen = strlen(log_path) + 1;\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||\n+\t !starts_with(key.mv_data, log_path))\n+\t\tret = 0;\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\n+\treturn ret;\n+}\n+\n+struct wrapped_each_ref_fn {\n+\teach_ref_fn *fn;\n+\tvoid *cb_data;\n+};\n+\n+static int check_reflog(const char *refname,\n+\t\t\tconst struct object_id *oid, int flags, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn *wrapped = cb_data;\n+\n+\tif (reflog_exists(refname))\n+\t\treturn wrapped->fn(refname, oid, 0, wrapped->cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn wrapped = {fn, cb_data};\n+\tint result = head_ref(fn, cb_data);\n+\tif (result)\n+\t\treturn result;\n+\treturn for_each_ref(check_reflog, &wrapped);\n+}\n+\n+static void strbuf_reflog_header(struct strbuf *sb, const char *refname)\n+{\n+\tuint64_t zero = 0;\n+\n+\tstrbuf_addf(sb, \"logs/%s\", refname);\n+\tstrbuf_add(sb, &zero, 8);\n+}\n+\n+static int lmdb_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+{\n+\t/*\n+\t * We mark that there is a reflog by creating a key of the\n+\t * form logs/$refname followed by nine \\0 (one for\n+\t * string-termination, 8 in lieu of a timestamp), with an empty\n+\t * value.\n+\t */\n+\n+\tint in_transaction = in_write_transaction();\n+\tMDB_val key, val;\n+\tstruct strbuf key_buf = STRBUF_INIT;\n+\n+\tif (!force_create && !should_autocreate_reflog(refname))\n+\t\treturn 0;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tstrbuf_reflog_header(&key_buf, refname);\n+\tkey.mv_size = key_buf.len + 1;\n+\tkey.mv_data = key_buf.buf;\n+\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\n+\tstrbuf_release(&key_buf);\n+\tif (!in_transaction)\n+\t\treturn mdb_transaction_commit(&transaction, err);\n+\treturn 0;\n+}\n+\n+struct expire_reflog_cb {\n+\tunsigned int flags;\n+\treflog_expiry_should_prune_fn *should_prune_fn;\n+\tvoid *policy_cb;\n+\tunsigned char last_kept_sha1[20];\n+};\n+\n+static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\tstruct expire_reflog_cb *cb = cb_data;\n+\tstruct expire_reflog_policy_cb *policy_cb = cb->policy_cb;\n+\n+\tif (cb->flags & EXPIRE_REFLOGS_REWRITE)\n+\t\tosha1 = cb->last_kept_sha1;\n+\n+\tif ((*cb->should_prune_fn)(osha1, nsha1, email, timestamp, tz,\n+\t\t\t\t message, policy_cb)) {\n+\t\tif (cb->flags & EXPIRE_REFLOGS_DRY_RUN)\n+\t\t\tprintf(\"would prune %s\", message);\n+\t\telse {\n+\t\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\t\tprintf(\"prune %s\", message);\n+\n+\t\t\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\t\t}\n+\t} else {\n+\t\thashcpy(cb->last_kept_sha1, nsha1);\n+\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\tprintf(\"keep %s\", message);\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_ref(const char *refname, const unsigned char *sha1)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction;\n+\n+\ttransaction = ref_transaction_begin(&err);\n+\tif (!transaction) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (ref_transaction_update(transaction, refname, sha1, NULL,\n+\t\t\t\t REF_NO_REFLOG, NULL, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (ref_transaction_commit(transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int lmdb_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\tstruct expire_reflog_cb cb;\n+\tint dry_run = flags & EXPIRE_REFLOGS_DRY_RUN;\n+\tint status = 0;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char resolved_sha1[20];\n+\tint type;\n+\tchar *resolved;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.reflog_expire(refname, sha1, flags, prepare_fn,\n+\t\t\t\t\t should_prune_fn, cleanup_fn,\n+\t\t\t\t\t policy_cb_data);\n+\n+\tmemset(&cb, 0, sizeof(cb));\n+\tcb.flags = flags;\n+\tcb.policy_cb = policy_cb_data;\n+\tcb.should_prune_fn = should_prune_fn;\n+\n+\tlmdb_transaction_begin_flags_or_die(dry_run ? MDB_RDONLY : 0);\n+\n+\tresolved = check_ref(transaction.txn, refname, sha1,\n+\t\t\t resolved_sha1, 0, &type);\n+\tif (!resolved)\n+\t\tdie(_(\"Failed to resolve %s\"), refname);\n+\tfree(resolved);\n+\n+\t(*prepare_fn)(refname, sha1, cb.policy_cb);\n+\tlmdb_for_each_reflog_ent(refname, expire_reflog_ent, &cb);\n+\t(*cleanup_fn)(cb.policy_cb);\n+\n+\tif (!dry_run) {\n+\t\t/*\n+\t\t * It doesn't make sense to adjust a reference pointed\n+\t\t * to by a symbolic ref based on expiring entries in\n+\t\t * the symbolic reference's reflog. Nor can we update\n+\t\t * a reference if there are no remaining reflog\n+\t\t * entries.\n+\t\t */\n+\t\tint update = (flags & EXPIRE_REFLOGS_UPDATE_REF) &&\n+\t\t\t!(type & REF_ISSYMREF) &&\n+\t\t\t!is_null_sha1(cb.last_kept_sha1);\n+\n+\t\tif (mdb_transaction_commit(&transaction, &err)) {\n+\t\t\tstatus |= error(_(\"couldn't write logs/%s: %s\"),\n+\t\t\t\t\trefname, err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t} else if (update && write_ref(refname, cb.last_kept_sha1)) {\n+\t\t\tstatus |= error(_(\"couldn't set %s\"),\n+\t\t\t\t\trefname);\n+\t\t}\n+\t}\n+\treturn status;\n+}\n+\n+static int lmdb_pack_refs(unsigned int flags)\n+{\n+\t/* This concept does not exist in this backend. */\n+\treturn 0;\n+}\n+\n+static int lmdb_peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\tint flag;\n+\tunsigned char base[20];\n+\n+\tif (read_ref_full(refname, RESOLVE_REF_READING, base, &flag))\n+\t\treturn -1;\n+\n+\treturn peel_object(base, sha1);\n+}\n+\n+static int lmdb_create_symref(const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg)\n+{\n+\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char old_sha1[20], new_sha1[20];\n+\tMDB_val key, val;\n+\tchar *valdata;\n+\tint ret = 0;\n+\tint in_transaction;\n+\n+\tin_transaction = in_write_transaction();\n+\n+\tif (logmsg && read_ref(ref_target, old_sha1))\n+\t\thashclr(old_sha1);\n+\n+\tkey.mv_size = strlen(ref_target) + 1;\n+\tkey.mv_data = xstrdup(ref_target);\n+\n+\tvaldata = xstrfmt(\"ref: %s\", refs_heads_master);\n+\tval.mv_data = valdata;\n+\tval.mv_size = strlen(valdata) + 1;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\n+\t/* TODO: Don't create ref d/f conflicts */\n+\n+\tif (logmsg && !read_ref(refs_heads_master, new_sha1) &&\n+\t log_ref_write(ref_target, old_sha1, new_sha1, logmsg, 0, &err)) {\n+\t\terror(_(\"create_symref: log_ref_write failed: %s\"), err.buf);\n+\t\tret = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (!in_transaction && mdb_transaction_commit(&transaction, &err)) {\n+\t\terror(_(\"create_symref: commit failed: %s\"), err.buf);\n+\t\tret = -1;\n+\t}\n+\n+done:\n+\tstrbuf_release(&err);\n+\tfree(key.mv_data);\n+\tfree(valdata);\n+\n+\treturn ret;\n+}\n+\n+MDB_env *submodule_txn_begin(struct lmdb_transaction *transaction)\n+{\n+\tint ret;\n+\tMDB_env *submodule_env = NULL;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_git_path_submodule(&path, transaction->submodule, \"refdb\");\n+\n+\tif (!is_directory(path.buf))\n+\t\tgoto done;\n+\n+\tmkdir(path.buf, 0775);\n+\n+\tinit_env(&submodule_env, path.buf);\n+\n+\tret = mdb_txn_begin(submodule_env, NULL, MDB_RDONLY, &transaction->txn);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_txn_begin failed: %s\", mdb_strerror(ret));\n+\n+\tret = mdb_dbi_open(transaction->txn, NULL, 0, &transaction->dbi);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_txn_open failed: %s\", mdb_strerror(ret));\n+\n+done:\n+\tstrbuf_release(&path);\n+\treturn submodule_env;\n+}\n+\n+static int lmdb_resolve_gitlink_ref(const char *submodule, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n+{\n+\tstruct lmdb_transaction transaction;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\ttransaction.txn = NULL;\n+\ttransaction.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn -1;\n+\tresult = !resolve_ref_unsafe_txn(&transaction, refname,\n+\t\t\t\t\t RESOLVE_REF_READING, sha1, NULL);\n+\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\n+\treturn result ? -1 : 0;\n+}\n+\n+static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (submodule) {\n+\t\tif (resolve_gitlink_ref(submodule, \"HEAD\", oid.hash) == 0)\n+\t\t\treturn fn(\"HEAD\", &oid, 0, cb_data);\n+\n+\t\treturn 0;\n+\t}\n+\n+\tif (!read_ref_full(\"HEAD\", RESOLVE_REF_READING, oid.hash, &flag))\n+\t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn do_head_ref(NULL, fn, cb_data);\n+}\n+\n+static int lmdb_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_head_ref(submodule, fn, cb_data);\n+}\n+\n+/*\n+ * Call fn for each reference for which the refname begins with base.\n+ * If trim is non-zero, then trim that many characters off the\n+ * beginning of each refname before passing the refname to fn. flags\n+ * can be DO_FOR_EACH_INCLUDE_BROKEN to include broken references in\n+ * the iteration. If fn ever returns a non-zero value, stop the\n+ * iteration and return that value; otherwise, return 0.\n+ */\n+static int do_for_each_ref(struct lmdb_transaction *transaction,\n+\t\t\t const char *base, each_ref_fn fn, int trim,\n+\t\t\t int flags, void *cb_data)\n+{\n+\n+\tMDB_val key, val;\n+\tMDB_cursor *cursor;\n+\tint baselen;\n+\tchar *search_key;\n+\tint retval;\n+\tint mdb_ret;\n+\n+\tretval = do_for_each_per_worktree_ref(transaction->submodule, base, fn,\n+\t\t\t\t\t trim, flags, cb_data);\n+\tif (retval)\n+\t\treturn retval;\n+\n+\tif (ref_paranoia < 0)\n+\t\tref_paranoia = git_env_bool(\"GIT_REF_PARANOIA\", 0);\n+\tif (ref_paranoia)\n+\t\tflags |= DO_FOR_EACH_INCLUDE_BROKEN;\n+\n+\tif (!base || !*base) {\n+\t\tbase = \"refs/\";\n+\t\ttrim = 0;\n+\t}\n+\n+\tsearch_key = xstrdup(base);\n+\tbaselen = strlen(base);\n+\tkey.mv_size = baselen + 1;\n+\tkey.mv_data = search_key;\n+\n+\tmdb_cursor_open_or_die(transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tstruct object_id oid;\n+\t\tint parsed_flags = 0;\n+\n+\t\tif (memcmp(key.mv_data, base, baselen))\n+\t\t\tbreak;\n+\n+\t\tparse_ref_data(transaction, (const char *)key.mv_data + (trim ? baselen : 0),\n+\t\t\t val.mv_data, oid.hash, 0, &parsed_flags, 0);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_BROKEN ||\n+\t\t (!(parsed_flags & REF_ISBROKEN) &&\n+\t\t has_sha1_file(oid.hash))) {\n+\t\t\tretval = fn((const char *)key.mv_data + (trim ? baselen : 0), &oid, parsed_flags, cb_data);\n+\t\t\tif (retval)\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\n+\treturn retval;\n+}\n+\n+static int lmdb_for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, \"\", fn, 0, 0, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct lmdb_transaction transaction;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref(fn, cb_data);\n+\n+\ttransaction.txn = NULL;\n+\ttransaction.submodule = submodule;\n+\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&transaction, \"\", fn, 0, 0, cb_data);\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, prefix, fn, strlen(prefix),\n+\t\t\t 0, cb_data);\n+}\n+\n+static int lmdb_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n+{\n+\tunsigned int flag = 0;\n+\n+\tif (broken)\n+\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, prefix, fn, 0, flag, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\tstruct lmdb_transaction transaction = {NULL};\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref_in(prefix, fn, cb_data);\n+\n+\ttransaction.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&transaction, prefix, fn,\n+\t\t\t\t strlen(prefix), 0, cb_data);\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, git_replace_ref_base, fn,\n+\t\t\t strlen(git_replace_ref_base), 0, cb_data);\n+}\n+\n+static int lmdb_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret;\n+\n+\tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tret = do_for_each_ref(&transaction, buf.buf, fn, 0, 0, cb_data);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, \"\", fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_read(const char *key)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup;\n+\tint ret;\n+\tint needs_free = 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_get_or_die(&transaction, &key_val, &val, &needs_free);\n+\tfree(keydup);\n+\tswitch (ret) {\n+\tcase 0:\n+\t\tprintf(\"%s\\n\", (char *)val.mv_data);\n+\t\treturn 0;\n+\tcase MDB_NOTFOUND:\n+\t\tfprintf(stderr, \"%s not found\\n\", key);\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 2;\n+\t}\n+\tif (needs_free)\n+\t\tfree(val.mv_data);\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_write(const char *key, const char *value)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup, *valdup;\n+\n+\tif (ref_type(key) != REF_TYPE_NORMAL) {\n+\t\tval.mv_data = (void *)value;\n+\t\tval.mv_size = strlen(value) + 1;\n+\t\twrite_per_worktree_ref(NULL, key, &val);\n+\t\treturn;\n+\t}\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tvaldup = xstrdup(value);\n+\tval.mv_data = valdup;\n+\tval.mv_size = strlen(value) + 1;\n+\n+\tmdb_put_or_die(&transaction, &key_val, &val, 0);\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\n+\tfree(keydup);\n+\tfree(valdup);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_delete(const char *key)\n+{\n+\tMDB_val key_val;\n+\tchar *keydup;\n+\tint ret;\n+\n+\tif (ref_type(key) != REF_TYPE_NORMAL)\n+\t\treturn del_per_worktree_ref(NULL, key, NULL);\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_del_or_die(&transaction, &key_val, NULL);\n+\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\n+\tfree(keydup);\n+\treturn ret;\n+}\n+\n+static int print_raw_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t\tconst char *email, unsigned long timestamp,\n+\t\t\t\tint tz, const char *message, void *cb_data)\n+{\n+\tint *any = cb_data;\n+\t*any = 1;\n+\n+\tif (*message != '\\n')\n+\t\tprintf(\"%s %s %s %lu %+05d\\t%s\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz, message);\n+\telse\n+\t\tprintf(\"%s %s %s %lu %+05d\\n\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz);\n+\treturn 0;\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_reflog(const char *refname)\n+{\n+\tint any = 0;\n+\n+\tfor_each_reflog_ent(refname, print_raw_reflog_ent, &any);\n+\n+\treturn !any;\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_delete_reflog(char *refname)\n+{\n+\tMDB_val key, val;\n+\tint mdb_ret;\n+\tchar *search_key;\n+\tMDB_cursor *cursor;\n+\tint len;\n+\n+\tsearch_key = xstrfmt(\"logs/%s\", refname ? refname : \"\");\n+\tlen = strlen(search_key) + 1;\n+\n+\tkey.mv_data = search_key;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\twhile (!mdb_ret) {\n+\t\tif (!starts_with(key.mv_data, search_key))\n+\t\t\tbreak;\n+\t\tif (refname && ((char *)val.mv_data)[len - 1] == 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\treturn;\n+}\n+\n+static void format_lmdb_reflog_ent(struct strbuf *dst, struct strbuf *src)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tconst char *msg;\n+\n+\tget_sha1_hex(src->buf, osha1);\n+\tget_sha1_hex(src->buf + 41, nsha1);\n+\n+\tmsg = strchr(src->buf + 82, '\\t');\n+\tif (msg)\n+\t\tmsg += 1;\n+\n+\tformat_reflog_entry(dst, osha1, nsha1, src->buf + 82, msg);\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_append_reflog(const char *refname)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tuint64_t now = getnanotime();\n+\tMDB_val key, val;\n+\tstruct strbuf key_buf = STRBUF_INIT;\n+\n+\tstrbuf_reflog_header(&key_buf, refname);\n+\tkey.mv_size = key_buf.len + 1;\n+\tkey.mv_data = key_buf.buf;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\t/* We do not remove the header entry here, because this is\n+\t * just for tests, so it's OK to be a bit inefficient */\n+\n+\twhile (strbuf_getline(&input, stdin) != EOF) {\n+\t\t/* \"logs/\" + \\0 + 8-byte timestamp for sorting and expiry */\n+\t\twrite_u64((char *)key.mv_data + key.mv_size - 8, htonll(now++));\n+\n+\t\t/*\n+\t\t * Convert the input from files-reflog format to\n+\t\t * lmdb-reflog-format\n+\t\t */\n+\n+\t\tformat_lmdb_reflog_ent(&sb, &input);\n+\t\tval.mv_data = sb.buf;\n+\t\tval.mv_size = sb.len + 1;\n+\t\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\t\tstrbuf_reset(&sb);\n+\t\tinput.len = 0;\n+\t}\n+\n+\tstrbuf_release(&input);\n+\tstrbuf_release(&sb);\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\tstrbuf_release(&key_buf);\n+}\n+\n+struct ref_storage_be refs_be_lmdb = {\n+\tNULL,\n+\t\"lmdb\",\n+\tlmdb_init_db,\n+\tlmdb_transaction_commit,\n+\tlmdb_transaction_commit, /* initial commit */\n+\n+\tlmdb_for_each_reflog_ent,\n+\tlmdb_for_each_reflog_ent_reverse,\n+\tlmdb_for_each_reflog,\n+\tlmdb_reflog_exists,\n+\tlmdb_create_reflog,\n+\tlmdb_delete_reflog,\n+\tlmdb_reflog_expire,\n+\n+\tlmdb_pack_refs,\n+\tlmdb_peel_ref,\n+\tlmdb_create_symref,\n+\tlmdb_delete_refs,\n+\tlmdb_rename_ref,\n+\n+\tlmdb_resolve_ref_unsafe,\n+\tlmdb_verify_refname_available,\n+\tlmdb_resolve_gitlink_ref,\n+\n+\tlmdb_head_ref,\n+\tlmdb_head_ref_submodule,\n+\tlmdb_for_each_ref,\n+\tlmdb_for_each_ref_submodule,\n+\tlmdb_for_each_ref_in,\n+\tlmdb_for_each_fullref_in,\n+\tlmdb_for_each_ref_in_submodule,\n+\tlmdb_for_each_rawref,\n+\tlmdb_for_each_namespaced_ref,\n+\tlmdb_for_each_replace_ref,\n+};\ndiff --git a/setup.c b/setup.c\nindex 1a62277..a47a5b8 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -279,7 +279,7 @@ int ref_storage_backend_config(const char *var, const char *value, void *ptr)\n *\n * - either an objects/ directory _or_ the proper\n * GIT_OBJECT_DIRECTORY environment variable\n- * - a refs/ directory\n+ * - a refdb/ directory or a refs/ directory\n * - either a HEAD symlink or a HEAD file that is formatted as\n * a proper \"ref:\", or a regular file HEAD that has a properly\n * formatted sha1 object name.\n@@ -313,8 +313,13 @@ int is_git_directory(const char *suspect)\n \n \tstrbuf_setlen(&path, len);\n \tstrbuf_addstr(&path, \"/refs\");\n-\tif (access(path.buf, X_OK))\n-\t\tgoto done;\n+\n+\tif (access(path.buf, X_OK)) {\n+\t\tstrbuf_setlen(&path, len);\n+\t\tstrbuf_addstr(&path, \"/refdb\");\n+\t\tif (access(path.buf, X_OK))\n+\t\t\tgoto done;\n+\t}\n \n \tret = 1;\n done:\ndiff --git a/test-refs-lmdb-backend.c b/test-refs-lmdb-backend.c\nnew file mode 100644\nindex 0000000..5cf61e6\n--- /dev/null\n+++ b/test-refs-lmdb-backend.c\n@@ -0,0 +1,64 @@\n+#include \"cache.h\"\n+#include \"string-list.h\"\n+#include \"parse-options.h\"\n+#include \"refs.h\"\n+\n+static const char * const test_refs_be_lmdb_usage[] = {\n+\t\"git test-refs-lmdb-backend \",\n+\t\"git test-refs-lmdb-backend \",\n+\tNULL,\n+};\n+\n+int test_refdb_raw_read(const char *key);\n+void test_refdb_raw_write(const char *key, const char *value);\n+int test_refdb_raw_reflog(const char *refname);\n+int test_refdb_raw_delete(const char *key);\n+void test_refdb_raw_delete_reflog(const char *refname);\n+void test_refdb_raw_append_reflog(const char *refname);\n+\n+int main(int argc, const char **argv)\n+{\n+\tconst char *delete = NULL;\n+\tconst char *reflog = NULL;\n+\tconst char *append_reflog = NULL;\n+\tint delete_missing_error = 0;\n+\tint clear_reflog = 0;\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING('d', NULL, &delete, \"branch\", \"delete refdb entry\"),\n+\t\tOPT_STRING('l', NULL, &reflog, \"branch\", \"show reflog\"),\n+\t\tOPT_STRING('a', NULL, &append_reflog, \"branch\", \"append to reflog\"),\n+\t\tOPT_BOOL('c', NULL, &clear_reflog, \"delete reflog. If a branch is provided, the reflog for that branch will be deleted; else all reflogs will be deleted.\"),\n+\t\tOPT_BOOL('x', NULL, &delete_missing_error,\n+\t\t\t \"deleting a missing key is an error\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, \"\", options, test_refs_be_lmdb_usage,\n+\t\t\t 0);\n+\n+\tif (!append_reflog && !clear_reflog && !delete && !reflog && argc != 1 && argc != 2)\n+\t\tusage_with_options(test_refs_be_lmdb_usage,\n+\t\t\t\t options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tregister_ref_storage_backend(&refs_be_lmdb);\n+\tset_ref_storage_backend(\"lmdb\");\n+\n+\tif (clear_reflog) {\n+\t\ttest_refdb_raw_delete_reflog(argv[0]);\n+\t} else if (append_reflog) {\n+\t\ttest_refdb_raw_append_reflog(append_reflog);\n+\t} else if (reflog) {\n+\t\treturn test_refdb_raw_reflog(reflog);\n+\t} else if (delete) {\n+\t\tif (test_refdb_raw_delete(delete) && delete_missing_error)\n+\t\t\treturn 1;\n+\t} else if (argc == 1) {\n+\t\treturn test_refdb_raw_read(argv[0]);\n+\t} else {\n+\t\ttest_refdb_raw_write(argv[0], argv[1]);\n+\t}\n+\treturn 0;\n+}\ndiff --git a/transport.c b/transport.c\nindex c92f8ae..67fbdcf 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1089,8 +1089,11 @@ static int refs_from_alternate_cb(struct alternate_object_database *e,\n \t\tgoto out;\n \t/* Is this a git repository with refs? */\n \tmemcpy(other + len - 8, \"/refs\", 6);\n-\tif (!is_directory(other))\n-\t\tgoto out;\n+\tif (!is_directory(other)) {\n+\t\tmemcpy(other + len - 8, \"/refdb\", 7);\n+\t\tif (!is_directory(other))\n+\t\t\tgoto out;\n+\t}\n \tother[len - 8] = '\\0';\n \tremote = remote_get(other);\n \ttransport = transport_get(remote, other);\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277586","messageId":"1454701462-3817-22-git-send-email-dturner@twopensource.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v4 21/21] refs: tests for lmdb backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-05T19:44:22Z","receivedAt":"2016-02-05T19:44:22Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add tests for the database backend.\n\nSigned-off-by: David Turner \nHelped-by: Dennis Kaarsemaker \n---\n t/t1460-refs-lmdb-backend.sh | 1109 +++++++++++++++++++++++++++++++++++\n t/t1470-refs-lmdb-backend-reflog.sh | 359 ++++++++++++\n t/t1480-refs-lmdb-submodule.sh | 85 +++\n t/test-lib.sh | 1 +\n 4 files changed, 1554 insertions(+)\n create mode 100755 t/t1460-refs-lmdb-backend.sh\n create mode 100755 t/t1470-refs-lmdb-backend-reflog.sh\n create mode 100755 t/t1480-refs-lmdb-submodule.sh\n\ndiff --git a/t/t1460-refs-lmdb-backend.sh b/t/t1460-refs-lmdb-backend.sh\nnew file mode 100755\nindex 0000000..31442e7\n--- /dev/null\n+++ b/t/t1460-refs-lmdb-backend.sh\n@@ -0,0 +1,1109 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2015 Twitter, Inc\n+# Copyright (c) 2006 Shawn Pearce\n+# This test is based on t1400-update-ref.sh\n+#\n+\n+test_description='Test lmdb refs backend'\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+if ! test_have_prereq LMDB\n+then\n+\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n+\ttest_done\n+fi\n+\n+raw_ref() {\n+\ttest-refs-lmdb-backend \"$1\"\n+}\n+\n+delete_ref() {\n+\ttest-refs-lmdb-backend -d \"$1\"\n+}\n+\n+write_ref() {\n+\ttest-refs-lmdb-backend \"$1\" \"$2\"\n+}\n+\n+raw_reflog() {\n+\ttest-refs-lmdb-backend -l \"$1\"\n+}\n+\n+delete_all_reflogs() {\n+\ttest-refs-lmdb-backend -c\n+}\n+\n+append_reflog() {\n+\ttest-refs-lmdb-backend -a \"$1\"\n+}\n+\n+Z=$_z40\n+\n+test_expect_success setup '\n+\tgit init --ref-storage=lmdb &&\n+\tfor name in A B C D E F\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tsha1=$(echo $name | git commit-tree $T) &&\n+\t\teval $name=$sha1\n+\tdone\n+'\n+\n+m=refs/heads/master\n+n_dir=refs/heads/gu\n+n=$n_dir/fixes\n+\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m with stale ref\" '\n+\ttest_must_fail git update-ref -d $m $A &&\n+\ttest $B = \"$(raw_ref $m)\"\n+'\n+test_expect_success \"delete $m\" '\n+\tgit update-ref -d $m $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \"delete $m without oldvalue verification\" \"\n+\tgit update-ref $m $A &&\n+\ttest $A = \\$(raw_ref $m) &&\n+\tgit update-ref -d $m &&\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"fail to create $n\" \\\n+\t\"git update-ref $n_dir $A &&\n+\t test_must_fail git update-ref $n $A >out 2>err\"\n+\n+delete_ref $n_dir\n+rm -f out err\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m (by HEAD) with stale ref\" '\n+\ttest_must_fail git update-ref -d HEAD $A &&\n+\ttest $B = $(raw_ref '\"$m\"')\n+'\n+test_expect_success \"delete $m (by HEAD)\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"pack refs\" \\\n+\t\"git pack-refs --all\"\n+test_expect_success \\\n+\t\"move $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"delete $m (by HEAD) should remove both packed and loose $m\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+OLD_HEAD=$(raw_ref HEAD)\n+test_expect_success \"delete symref without dereference\" '\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+\n+test_expect_success \"delete symref without dereference when the referred ref is packed\" '\n+\techo foo >foo.c &&\n+\tgit add foo.c &&\n+\tgit commit -m foo &&\n+\tgit pack-refs --all &&\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+delete_ref $m\n+\n+test_expect_success 'update-ref -d is not confused by self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\ttest_must_fail git update-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\tgit update-ref --no-deref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '\n+\ttest-refs-lmdb-backend refs/heads/bad \"\" &&\n+\ttest_when_finished \"delete_ref refs/heads/bad\" &&\n+\tgit symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&\n+\ttest_when_finished \"delete_ref refs/heads/ref-to-bad\" &&\n+\traw_ref refs/heads/ref-to-bad &&\n+\tgit update-ref --no-deref -d refs/heads/ref-to-bad &&\n+\t! raw_ref refs/heads/ref-to-bad\n+'\n+\n+test_expect_success '(not) create HEAD with old sha1' \"\n+\ttest_must_fail git update-ref HEAD $A $B\n+\"\n+test_expect_success \"(not) prior created .git/$m\" \"\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create HEAD\" \\\n+\t\"git update-ref HEAD $A\"\n+test_expect_success '(not) change HEAD with wrong SHA1' \"\n+\ttest_must_fail git update-ref HEAD $B $Z\n+\"\n+test_expect_success \"(not) changed .git/$m\" \"\n+\t! test $B\"' = $(raw_ref '\"$m\"')\n+'\n+\n+: a repository with working tree always has reflog these days...\n+delete_all_reflogs\n+: | append_reflog $m\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:30\" \\\n+\t git update-ref HEAD '\"$A\"' -m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:31\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:41\" \\\n+\t git update-ref HEAD'\" $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150200 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t\"raw_reflog $m >actual &&\n+\t test_cmp expect actual\"\n+delete_ref $m\n+delete_all_reflogs\n+: | append_reflog $m\n+rm -f actual expect\n+\n+test_expect_success \\\n+\t'enable core.logAllRefUpdates' \\\n+\t'git config core.logAllRefUpdates true &&\n+\t test true = $(git config --bool --get core.logAllRefUpdates)'\n+\n+test_expect_success \\\n+\t\"create $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:32\" \\\n+\t git update-ref HEAD'\" $A \"'-m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:33\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:43\" \\\n+\t git update-ref HEAD '\"$A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150320 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t'raw_reflog $m >actual &&\n+\ttest_cmp expect actual'\n+delete_ref $m\n+rm -f expect\n+\n+git update-ref $m $D\n+git reflog expire --expire=all $m\n+\n+append_reflog $m < 1117150320 -0500\n+$C $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150350 -0500\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 -0500\n+$F $Z $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150680 -0500\n+$Z $E $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 -0500\n+EOF\n+\n+ed=\"Thu, 26 May 2005 18:32:00 -0500\"\n+gd=\"Thu, 26 May 2005 18:33:00 -0500\"\n+ld=\"Thu, 26 May 2005 18:43:00 -0500\"\n+test_expect_success \\\n+\t'Query \"master@{May 25 2005}\" (before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 25 2005}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t\"Query master@{2005-05-25} (before history)\" \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify master@{2005-05-25} >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t echo test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:31:59}\" (1 second before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:31:59}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\''master'\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:00}\" (exactly history start)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:00}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:30}\" (first non-creation change)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:30}\" >o 2>e &&\n+\t test '\"$A\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:33:01}\" (middle of history with gap)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:33:01}\" >o 2>e &&\n+\t test '\"$B\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m has gap after $gd\"'.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:38:00}\" (middle of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:38:00}\" >o 2>e &&\n+\t test '\"$Z\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:43:00}\" (exact end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:43:00}\" >o 2>e &&\n+\t test '\"$E\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-28}\" (past end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-28}\" >o 2>e &&\n+\t test '\"$D\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m unexpectedly ended on $ld\"'.\" = \"$(cat e)\"'\n+\n+\n+git reflog expire --expire=all $m\n+delete_ref $m\n+\n+test_expect_success \\\n+ 'creating initial files' \\\n+ 'echo TEST >F &&\n+ git add F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:30\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:30\" git commit -m add -a &&\n+\t h_TEST=$(git rev-parse --verify HEAD) &&\n+\t echo The other day this did not work. >M &&\n+\t echo And then Bob told me how to fix it. >>M &&\n+\t echo OTHER >F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:41\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:41\" git commit -F M -a &&\n+\t h_OTHER=$(git rev-parse --verify HEAD) &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:44\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:44\" git commit --amend &&\n+\t h_FIXED=$(git rev-parse --verify HEAD) &&\n+\t echo Merged initial commit and a later commit. >M &&\n+\t echo $h_TEST >.git/MERGE_HEAD &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:45\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:45\" git commit -F M &&\n+\t h_MERGED=$(git rev-parse --verify HEAD) &&\n+\t rm -f M'\n+\n+cat >expect < 1117150200 +0000\tcommit (initial): add\n+$h_TEST $h_OTHER $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\tcommit: The other day this did not work.\n+$h_OTHER $h_FIXED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151040 +0000\tcommit (amend): The other day this did not work.\n+$h_FIXED $h_MERGED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151100 +0000\tcommit (merge): Merged initial commit and a later commit.\n+EOF\n+test_expect_success \\\n+\t'git commit logged updates' \\\n+\t\"raw_reflog $m >actual &&\n+\ttest_cmp expect actual\"\n+unset h_TEST h_OTHER h_FIXED h_MERGED\n+\n+test_expect_success \\\n+\t'git cat-file blob master:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob master:F)'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:30}:F (expect TEST)' \\\n+\t'test TEST = $(git cat-file blob \"master@{2005-05-26 23:30}:F\")'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:42}:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob \"master@{2005-05-26 23:42}:F\")'\n+\n+a=refs/heads/a\n+b=refs/heads/b\n+c=refs/heads/c\n+E='\"\"'\n+F='%s\\0'\n+pws='path with space'\n+\n+test_expect_success 'stdin test setup' '\n+\techo \"$pws\" >\"$pws\" &&\n+\tgit add -- \"$pws\" &&\n+\tgit commit -m \"$pws\"\n+'\n+\n+test_expect_success '-z fails without --stdin' '\n+\ttest_must_fail git update-ref -z $m $m $m 2>err &&\n+\tgrep \"usage: git update-ref\" err\n+'\n+\n+test_expect_success 'stdin works with no input' '\n+\t>stdin &&\n+\tgit update-ref --stdin stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin fails on only whitespace' '\n+\techo \" \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin fails on leading whitespace' '\n+\techo \" create $a $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a $m\" err\n+'\n+\n+test_expect_success 'stdin fails on unknown command' '\n+\techo \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin fails on unbalanced quotes' '\n+\techo \"create $a \\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails on invalid escape' '\n+\techo \"create $a \\\"ma\\zter\\\"\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"ma\\\\\\\\zter\\\\\\\"\" err\n+'\n+\n+test_expect_success 'stdin fails on junk after quoted argument' '\n+\techo \"create \\\"$a\\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unexpected character after quoted argument: \\\\\\\"$a\\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails create with no ref' '\n+\techo \"create \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with no new value' '\n+\techo \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with too many arguments' '\n+\techo \"create $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails update with no ref' '\n+\techo \"update \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with no new value' '\n+\techo \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with too many arguments' '\n+\techo \"update $a $m $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails delete with no ref' '\n+\techo \"delete \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin fails delete with too many arguments' '\n+\techo \"delete $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails verify with too many arguments' '\n+\techo \"verify $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: verify $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails option with unknown name' '\n+\techo \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin fails with duplicate refs' '\n+\tcat >stdin <<-EOF &&\n+\tcreate $a $m\n+\tcreate $b $m\n+\tcreate $a $m\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin create ref works' '\n+\techo \"create $a $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with quoted argument' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"$m\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with escaped character' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"ma\\\\163ter\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update ref creates with zero old value' '\n+\techo \"update $b $m $Z\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin update ref creates with empty old value' '\n+\techo \"update $b $m $E\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin create ref works with path with space to blob' '\n+\techo \"create refs/blobs/pws \\\"$m:$pws\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin update ref fails with wrong old value' '\n+\techo \"update $c $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref fails with bad old value' '\n+\techo \"update $c $m does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with bad new value' '\n+\techo \"create $c does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with zero new value' '\n+\techo \"create $c \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: zero \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref works with right old value' '\n+\techo \"update $b $m~1 $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with wrong old value' '\n+\techo \"delete $a $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with zero old value' '\n+\techo \"delete $a \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tupdate refs/TESTSYMREF $a $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tdelete refs/TESTSYMREF $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref works with right old value' '\n+\techo \"delete $b $m~1\" >stdin &&\n+\tgit update-ref --stdin stdin <<-EOF &&\n+\tupdate $a $m\n+\tcreate $b $m\n+\tverify $c\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $m\" >stdin &&\n+\tgit update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify succeeds for missing reference' '\n+\techo \"verify refs/heads/missing $Z\" >stdin &&\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref --stdin expect &&\n+\techo \"verify $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $Z\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update refs works with identity updates' '\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c $Z $E\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c ''\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tcat >stdin <<-EOF &&\n+\tdelete $a $m\n+\tupdate $b $Z $m\n+\tupdate $c $E $m~1\n+\tEOF\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on empty command' '\n+\tprintf $F \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin -z fails on only whitespace' '\n+\tprintf $F \" \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on leading whitespace' '\n+\tprintf $F \" create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails on unknown command' '\n+\tprintf $F \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no ref' '\n+\tprintf $F \"create \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no new value' '\n+\tprintf $F \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with too many arguments' '\n+\tprintf $F \"create $a\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no ref' '\n+\tprintf $F \"update \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too few args' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z emits warning with empty new value' '\n+\tgit update-ref $a $m &&\n+\tprintf $F \"update $a\" \"\" \"\" >stdin &&\n+\tgit update-ref -z --stdin err &&\n+\tgrep \"warning: update $a: missing , treating as zero\" err &&\n+\ttest_must_fail git rev-parse --verify -q $a\n+'\n+\n+test_expect_success 'stdin -z fails update with no new value' '\n+\tprintf $F \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no old value' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too many arguments' '\n+\tprintf $F \"update $m\" \"$m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no ref' '\n+\tprintf $F \"delete \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no old value' '\n+\tprintf $F \"delete $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with too many arguments' '\n+\tprintf $F \"delete $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with too many arguments' '\n+\tprintf $F \"verify $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with no old value' '\n+\tprintf $F \"verify $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: verify $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails option with unknown name' '\n+\tprintf $F \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin -z fails with duplicate refs' '\n+\tprintf $F \"create $a\" \"$m\" \"create $b\" \"$m\" \"create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin -z create ref works' '\n+\tprintf $F \"create $a\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update ref creates with zero old value' '\n+\tprintf $F \"update $b\" \"$m\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin -z update ref creates with empty old value' '\n+\tprintf $F \"update $b\" \"$m\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref works with path with space to blob' '\n+\tprintf $F \"create refs/blobs/pws\" \"$m:$pws\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin -z update ref fails with wrong old value' '\n+\tprintf $F \"update $c\" \"$m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref fails with bad old value' '\n+\tprintf $F \"update $c\" \"$m\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails when ref exists' '\n+\tgit update-ref $c $m &&\n+\tgit rev-parse \"$c\" >expect &&\n+\tprintf $F \"create $c\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse \"$c\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref fails with bad new value' '\n+\tgit update-ref -d \"$c\" &&\n+\tprintf $F \"create $c\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails with empty new value' '\n+\tprintf $F \"create $c\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: missing \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref works with right old value' '\n+\tprintf $F \"update $b\" \"$m~1\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with wrong old value' '\n+\tprintf $F \"delete $a\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with zero old value' '\n+\tprintf $F \"delete $a\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"update refs/TESTSYMREF\" \"$a\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"delete refs/TESTSYMREF\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref works with right old value' '\n+\tprintf $F \"delete $b\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for missing reference' '\n+\tprintf $F \"verify refs/heads/missing\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tprintf $F \"verify $m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update refs works with identity updates' '\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$Z\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tprintf $F \"delete $a\" \"$m\" \"update $b\" \"$Z\" \"$m\" \"update $c\" \"\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin /dev/null || test-refs-lmdb-backend -l \"$1\"\n+}\n+\n+append_reflog() {\n+\ttest-refs-lmdb-backend -a \"$1\"\n+}\n+\n+check_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\" && git cat-file -t $o || {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone &&\n+\ttest -z \"$gaah\"\n+}\n+\n+check_fsck () {\n+\toutput=$(git fsck --full)\n+\tcase \"$1\" in\n+\t'')\n+\t\ttest -z \"$output\" ;;\n+\t*)\n+\t\techo \"$output\" | grep \"$1\" ;;\n+\tesac\n+}\n+\n+corrupt () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmv .git/objects/$aa/$zz .git/$aa$zz\n+}\n+\n+recover () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmkdir -p .git/objects/$aa\n+\tmv .git/$aa$zz .git/objects/$aa/$zz\n+}\n+\n+check_dont_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\"\n+\t\tgit cat-file -t $o && {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone\n+\ttest -z \"$gaah\"\n+}\n+\n+test_expect_success setup '\n+\tgit init --ref-storage=lmdb &&\n+\tmkdir -p A/B &&\n+\techo rat >C &&\n+\techo ox >A/D &&\n+\techo tiger >A/B/E &&\n+\tgit add . &&\n+\n+\ttest_tick && git commit -m rabbit &&\n+\tH=`git rev-parse --verify HEAD` &&\n+\tA=`git rev-parse --verify HEAD:A` &&\n+\tB=`git rev-parse --verify HEAD:A/B` &&\n+\tC=`git rev-parse --verify HEAD:C` &&\n+\tD=`git rev-parse --verify HEAD:A/D` &&\n+\tE=`git rev-parse --verify HEAD:A/B/E` &&\n+\tcheck_fsck &&\n+\n+\ttest_chmod +x C &&\n+\tgit add C &&\n+\ttest_tick && git commit -m dragon &&\n+\tL=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f C A/B/E &&\n+\techo snake >F &&\n+\techo horse >A/G &&\n+\tgit add F A/G &&\n+\ttest_tick && git commit -a -m sheep &&\n+\tF=`git rev-parse --verify HEAD:F` &&\n+\tG=`git rev-parse --verify HEAD:A/G` &&\n+\tI=`git rev-parse --verify HEAD:A` &&\n+\tJ=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f A/G &&\n+\ttest_tick && git commit -a -m monkey &&\n+\tK=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tcheck_fsck &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success rewind '\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\ttest -f C &&\n+\ttest -f A/B/E &&\n+\t! test -f F &&\n+\t! test -f A/G &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog\n+'\n+\n+test_expect_success 'corrupt and check' '\n+\n+\tcorrupt $F &&\n+\tcheck_fsck \"missing blob $F\"\n+\n+'\n+\n+test_expect_success 'reflog expire --dry-run should not touch reflog' '\n+\n+\tgit reflog expire --dry-run \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog &&\n+\n+\tcheck_fsck \"missing blob $F\"\n+'\n+\n+test_expect_success 'reflog expire' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\techo git reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 2 reflog &&\n+\n+\tcheck_fsck \"dangling commit $K\"\n+'\n+\n+test_expect_success 'prune and fsck' '\n+\n+\tgit prune &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E H L &&\n+\tcheck_dont_have F G I J K\n+\n+'\n+\n+test_expect_success 'recover and check' '\n+\n+\trecover $F &&\n+\tcheck_fsck \"dangling blob $F\"\n+\n+'\n+\n+test_expect_success 'delete' '\n+\techo 1 > C &&\n+\ttest_tick &&\n+\tgit commit -m rat C &&\n+\n+\techo 2 > C &&\n+\ttest_tick &&\n+\tgit commit -m ox C &&\n+\n+\techo 3 > C &&\n+\ttest_tick &&\n+\tgit commit -m tiger C &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\tmaster_entry_count=$(git reflog show master | wc -l) &&\n+\n+\ttest $HEAD_entry_count = 5 &&\n+\ttest $master_entry_count = 5 &&\n+\n+\n+\tgit reflog delete master@{1} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\ttest $HEAD_entry_count = $(git reflog | wc -l) &&\n+\t! grep ox < output &&\n+\n+\tmaster_entry_count=$(wc -l < output) &&\n+\n+\tgit reflog delete HEAD@{1} &&\n+\ttest $(($HEAD_entry_count -1)) = $(git reflog | wc -l) &&\n+\ttest $master_entry_count = $(git reflog show master | wc -l) &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\n+\tgit reflog delete master@{07.04.2005.15:15:00.-0700} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\t! grep dragon < output\n+\n+'\n+\n+test_expect_success 'rewind2' '\n+\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success '--expire=never' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=never \\\n+\t\t--expire-unreachable=never \\\n+\t\t--all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=never' '\n+\n+\tgit config gc.reflogexpire never &&\n+\tgit config gc.reflogexpireunreachable never &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=false' '\n+\n+\tgit config gc.reflogexpire false &&\n+\tgit config gc.reflogexpireunreachable false &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog &&\n+\n+\tgit config --unset gc.reflogexpire &&\n+\tgit config --unset gc.reflogexpireunreachable\n+\n+'\n+\n+test_expect_success 'checkout should not delete log for packed ref' '\n+\ttest $(git reflog master | wc -l) = 4 &&\n+\tgit branch foo &&\n+\tgit pack-refs --all &&\n+\tgit checkout foo &&\n+\ttest $(git reflog master | wc -l) = 4\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs on)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# now logs/refs/heads/one is a stale directory, but\n+\t# we should move it out of the way to create \"one\" reflog\n+\tgit branch one master &&\n+\techo \"one@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs off)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# same as before, but we only create a reflog for \"one\" if\n+\t# it already exists, which it does not\n+\tgit -c core.logallrefupdates=false branch one master &&\n+\t: >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Triggering the bug detected by this test requires a newline to fall\n+# exactly BUFSIZ-1 bytes from the end of the file. We don't know\n+# what that value is, since it's platform dependent. However, if\n+# we choose some value N, we also catch any D which divides N evenly\n+# (since we will read backwards in chunks of D). So we choose 8K,\n+# which catches glibc (with an 8K BUFSIZ) and *BSD (1K).\n+#\n+# Each line is 114 characters, so we need 75 to still have a few before the\n+# last 8K. The 89-character padding on the final entry lines up our\n+# newline exactly.\n+test_expect_success 'parsing reverse reflogs at BUFSIZ boundaries' '\n+\tgit checkout -b reflogskip &&\n+\tz38=00000000000000000000000000000000000000 &&\n+\tident=\"abc 0000000001 +0000\" &&\n+\tfor i in $(test_seq 1 75); do\n+\t\tprintf \"$z38%02d $z38%02d %s\\t\" $i $(($i+1)) \"$ident\" &&\n+\t\tif test $i = 75; then\n+\t\t\tfor j in $(test_seq 1 89); do\n+\t\t\t\tprintf X\n+\t\t\tdone\n+\t\telse\n+\t\t\tprintf X\n+\t\tfi &&\n+\t\tprintf \"\\n\"\n+\tdone | append_reflog refs/heads/reflogskip &&\n+\tgit rev-parse reflogskip@{73} >actual &&\n+\techo ${z38}03 >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1480-refs-lmdb-submodule.sh b/t/t1480-refs-lmdb-submodule.sh\nnew file mode 100755\nindex 0000000..c7d911a\n--- /dev/null\n+++ b/t/t1480-refs-lmdb-submodule.sh\n@@ -0,0 +1,85 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2016 Twitter, Inc\n+# Based on t5531-deep-submodule-push.sh\n+\n+test_description='Test lmdb refs backend'\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+if ! test_have_prereq LMDB\n+then\n+\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n+\ttest_done\n+fi\n+\n+test_expect_success setup '\n+\tmkdir pub.git &&\n+\tGIT_DIR=pub.git git init --bare &&\n+\tGIT_DIR=pub.git git config receive.fsckobjects true &&\n+\tmkdir work &&\n+\t(\n+\t\tcd work &&\n+\t\tgit init --ref-storage=lmdb &&\n+\t\tgit config push.default matching &&\n+\t\tmkdir -p gar/bage &&\n+\t\t(\n+\t\t\tcd gar/bage &&\n+\t\t\tgit init --ref-storage=lmdb &&\n+\t\t\tgit config push.default matching &&\n+\t\t\t>junk &&\n+\t\t\tgit add junk &&\n+\t\t\tgit commit -m \"Initial junk\"\n+\t\t) &&\n+\t\tgit add gar/bage &&\n+\t\tgit commit -m \"Initial superproject\"\n+\t)\n+'\n+\n+test_expect_success 'submodules have same ref storage' '\n+\tgit init --ref-storage=lmdb test &&\n+\t(\n+\t\tcd test &&\n+\t\tgit submodule add ../work/gar/bage w\n+\t) &&\n+\t(\n+\t\tcd test/w &&\n+\t\tgit config extensions.refstorage >cfg &&\n+\t\techo lmdb >expect &&\n+\t\ttest_cmp cfg expect\n+\t)\n+'\n+\n+test_expect_success 'push with correct backend' '\n+\t(\n+\t\tcd work/gar/bage &&\n+\t\t>junk2 &&\n+\t\tgit add junk2 &&\n+\t\tgit commit -m \"Second junk\"\n+\t) &&\n+\t(\n+\t\tcd work &&\n+\t\tgit add gar/bage &&\n+\t\tgit commit -m \"Second commit for gar/bage\" &&\n+\t\tgit push --recurse-submodules=check ../pub.git master\n+\t)\n+'\n+\n+test_expect_success 'commit with different backend fails' '\n+\t(\n+\t\tcd work/gar/bage &&\n+\t\ttest_commit junk3 &&\n+\t\t# manually convert to files-backend\n+\t\tgitdir=\"$(git rev-parse --git-dir)\" &&\n+\t\tmkdir -p \"$gitdir/refs/heads\" &&\n+\t\tgit rev-parse HEAD >\"$gitdir/refs/heads/master\" &&\n+\t\tgit config --local --unset extensions.refStorage &&\n+\t\trm -r \"$gitdir/refdb\"\n+\t) &&\n+\t(\n+\t\tcd work &&\n+\t\ttest_must_fail git add gar/bage\n+\t)\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0b47eb6..ce40770 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -960,6 +960,7 @@ test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE\" && test_set_prereq LIBPCRE\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$USE_LIBLMDB\" && test_set_prereq LMDB\n \n # Can we rely on git's output in the C locale?\n if test -n \"$GETTEXT_POISON\"\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"277789","messageId":"xmqq8u2u6aeu.fsf@gitster.mtv.corp.google.com","threadId":"41344","inReplyTo":"1454701462-3817-1-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 00/20] refs backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-08T23:37:45Z","receivedAt":"2016-02-08T23:37:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> Changes to this version:\n> re-rolled on top of pu as-of 9db66d9f1aa.\n>\n> Bug fixes include:\n> For submodules: memory leaks; segfault on bad config. (thanks to Peff)\n> In symref splitting: check that would always succeed (thanks to Peff)\n> A bogus double-declaration of a var (thanks to Ramsay Jones)\n> Two memory leaks (thanks to Thomas Gummerer)\n> An unused var (thanks to Duy Nguyen)\n>\n> Other improvements are:\n> Strings prepped for 18n (thanks to Duy Nguyen)\n> Cleaner submodule handling (thanks to Peff)\n> Whitelisting instead of blacklisting in git-new-workdir (thanks to\n> Thomas Gummerer)\n> Allow older gits to recognize lmdb-backend git repos (thanks to Duy Nguyen)\n> Tab completion and cleaner commit messages (thanks SZEDER Gábor)\n> Removed some #ifdefs, moving all backend setup to one place (thanks to Duy\n> Nguyen)\n>\n> Thanks to all for reviews.\n\nThanks for a reroll.\n\nApplied this on top of a merge between the current 'master' and\n'sb/submodule-parallel-update' topic to untangle the dependency;\notherwise there is no way for this topic to make progress X-<.\n\nLet's see how well this round goes.\n"},{"id":"277908","messageId":"56BC3C08.2080003@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-3-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 02/21] refs: add methods for misc ref operations","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T07:45:12Z","receivedAt":"2016-02-11T07:45:12Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> From: Ronnie Sahlberg \n> \n> Add ref backend methods for:\n> resolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,\n> create_symref, resolve_gitlink_ref.\n> \n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> builtin/init-db.c | 1 +\n> refs.c | 36 ++++++++++++++++++++++++++++++++++++\n> refs/files-backend.c | 33 +++++++++++++++++++++++----------\n> refs/refs-internal.h | 23 +++++++++++++++++++++++\n> 4 files changed, 83 insertions(+), 10 deletions(-)\n> \n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 07229d6..26e1cc3 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -8,6 +8,7 @@\n> #include \"builtin.h\"\n> #include \"exec_cmd.h\"\n> #include \"parse-options.h\"\n> +#include \"refs.h\"\n\nYou can't see it in this diff's context, but there is already an\n'#include \"refs.h\"' in this file.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277911","messageId":"56BC4985.5070707@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-4-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 03/21] refs: add methods for the ref iterators","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T08:42:45Z","receivedAt":"2016-02-11T08:42:45Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> From: Ronnie Sahlberg \n> \n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n> refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n> refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n> 3 files changed, 112 insertions(+), 12 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index afdde7d..e598b73 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1158,3 +1158,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n> {\n> \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n> }\n> +\n> +int head_ref(each_ref_fn fn, void *cb_data)\n> +{\n> +\treturn the_refs_backend->head_ref(fn, cb_data);\n> +}\n> +\n> +int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n> +{\n> +\treturn the_refs_backend->head_ref_submodule(submodule, fn, cb_data);\n> +}\n> +\n\nI think it is unnecessary to have so many virtual functions. For\nexample, here you have made head_ref_submodule() virtual. But the files\nand lmdb implementations of this function are identical. They both call\ndo_head_ref(), which are (confusingly) two independent static functions.\nBut those functions are *also* defined identically, namely\n\n> static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n> {\n> \tstruct object_id oid;\n> \tint flag;\n> \n> \tif (submodule) {\n> \t\tif (resolve_gitlink_ref(submodule, \"HEAD\", oid.hash) == 0)\n> \t\t\treturn fn(\"HEAD\", &oid, 0, cb_data);\n> \n> \t\treturn 0;\n> \t}\n> \n> \tif (!read_ref_full(\"HEAD\", RESOLVE_REF_READING, oid.hash, &flag))\n> \t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n> \n> \treturn 0;\n> }\n> \n\nAfter all, what else would they possibly want to do? And both\nresolve_gitlink_ref() and read_ref_full() are already virtual functions\n(actually read_ref_full() only has a single definition but it calls the\nvirtualized resolve_ref_unsafe()).\n\nSo it seems to me that it is unnecessary for head_ref_submodule() to be\nvirtual, and that there only needs to be one definition of do_head_ref().\n\n(Off-topic: for that matter, do_head_ref() is itself almost pointless.\nIt could easily be inlined if we were sure that nobody passes it\nsubmodule==NULL.)\n\nSimilarly, I bet that if do_for_each_ref() were just a little bit more\ncapable, then the following functions could also remain non-virtual\n(i.e., one definition could be shared across all backend implementations):\n\n* for_each_ref()\n* for_each_ref_submodule()\n* for_each_ref_in()\n* for_each_fullref_in()\n* for_each_ref_in_submodule()\n* for_each_replace_ref()\n* for_each_namespaced_ref()\n* for_each_rawref()\n\n> [...]\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index e83bc22..4a1d215 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -216,6 +216,24 @@ typedef int verify_refname_available_fn(const char *refname, struct string_list\n> typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n> \t\t\t\t unsigned char *sha1);\n> \n> +/* iteration methods */\n> +typedef int head_ref_fn(each_ref_fn fn, void *cb_data);\n> +typedef int head_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n> +\t\t\t\t void *cb_data);\n> +typedef int for_each_ref_fn(each_ref_fn fn, void *cb_data);\n> +typedef int for_each_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n> +\t\t\t\t void *cb_data);\n> +typedef int for_each_ref_in_fn(const char *prefix, each_ref_fn fn,\n> +\t\t\t void *cb_data);\n> +typedef int for_each_fullref_in_fn(const char *prefix, each_ref_fn fn,\n> +\t\t\t\t void *cb_data, unsigned int broken);\n> +typedef int for_each_ref_in_submodule_fn(const char *submodule,\n> +\t\t\t\t\t const char *prefix,\n> +\t\t\t\t\t each_ref_fn fn, void *cb_data);\n> +typedef int for_each_rawref_fn(each_ref_fn fn, void *cb_data);\n> +typedef int for_each_namespaced_ref_fn(each_ref_fn fn, void *cb_data);\n> +typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n> +\n> struct ref_storage_be {\n> \tstruct ref_storage_be *next;\n> \tconst char *name;\n> @@ -228,6 +246,17 @@ struct ref_storage_be {\n> \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n> \tverify_refname_available_fn *verify_refname_available;\n> \tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n> +\n> +\thead_ref_fn *head_ref;\n> +\thead_ref_submodule_fn *head_ref_submodule;\n> +\tfor_each_ref_fn *for_each_ref;\n> +\tfor_each_ref_submodule_fn *for_each_ref_submodule;\n> +\tfor_each_ref_in_fn *for_each_ref_in;\n> +\tfor_each_fullref_in_fn *for_each_fullref_in;\n> +\tfor_each_ref_in_submodule_fn *for_each_ref_in_submodule;\n> +\tfor_each_rawref_fn *for_each_rawref;\n> +\tfor_each_namespaced_ref_fn *for_each_namespaced_ref;\n> +\tfor_each_replace_ref_fn *for_each_replace_ref;\n> };\n> \n> extern struct ref_storage_be refs_be_files;\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277912","messageId":"56BC4AE4.6060708@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-21-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T08:48:36Z","receivedAt":"2016-02-11T08:48:36Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Add a database backend for refs using LMDB. This backend runs git\n> for-each-ref about 30% faster than the files backend with fully-packed\n> refs on a repo with ~120k refs. It's also about 4x faster than using\n> fully-unpacked refs. In addition, and perhaps more importantly, it\n> avoids case-conflict issues on OS X.\n\nWhen I compile this using gcc with -Werror=pointer-arith, I get the\nfollowing errors:\n\n> refs/lmdb-backend.c: In function ‘rename_reflog_ent’:\n> refs/lmdb-backend.c:1068:39: error: pointer of type ‘void *’ used in arithmetic [-Werror=pointer-arith]\n> strbuf_add(&new_key_buf, key.mv_data + key.mv_size - 8, 8);\n> ^\n> refs/lmdb-backend.c:1068:53: error: pointer of type ‘void *’ used in arithmetic [-Werror=pointer-arith]\n> strbuf_add(&new_key_buf, key.mv_data + key.mv_size - 8, 8);\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277915","messageId":"56BC4C3D.8040505@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-9-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 08/21] refs: add methods to init refs db","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T08:54:21Z","receivedAt":"2016-02-11T08:54:21Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Alternate refs backends might not need the refs/heads directory and so\n> on, so we make ref db initialization part of the backend.\n> \n> Signed-off-by: David Turner \n> ---\n> builtin/init-db.c | 20 ++++++++++----------\n> refs.c | 5 +++++\n> refs.h | 2 ++\n> refs/files-backend.c | 16 ++++++++++++++++\n> refs/refs-internal.h | 2 ++\n> 5 files changed, 35 insertions(+), 10 deletions(-)\n> \n> [...]\n> diff --git a/refs.h b/refs.h\n> index 5bc3fbc..c5ecc52 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -66,6 +66,8 @@ extern int ref_exists(const char *refname);\n> \n> extern int is_branch(const char *refname);\n> \n> +extern int refs_init_db(struct strbuf *err, int shared);\n\nThe convention elsewhere is for \"struct strbuf *err\" to be the *last*\nparameter.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277914","messageId":"56BC4D94.4040003@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-10-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 09/21] refs: add method to rename refs","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T09:00:04Z","receivedAt":"2016-02-11T09:00:04Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Signed-off-by: David Turner \n> ---\n> refs.c | 5 +++++\n> refs/files-backend.c | 4 +++-\n> refs/refs-internal.h | 9 +++++++++\n> 3 files changed, 17 insertions(+), 1 deletion(-)\n> \n> [...]\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 5768fee..15fa99c 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -67,6 +67,13 @@ int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n> \t\t\t\t each_ref_fn fn, int trim, int flags,\n> \t\t\t\t void *cb_data);\n> \n> +/*\n> + * Check if the new name does not conflict with any existing refs\n> + * (other than possibly the old ref). Return 0 if the ref can be\n> + * renamed to the new name.\n> + */\n> +int rename_ref_available(const char *oldname, const char *newname);\n\nThanks for adding a docstring for this function. But this function\nalready has a declaration later in the file so you should remove one or\nthe other.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277916","messageId":"56BC5C8C.8060201@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-13-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 12/21] refs: allow log-only updates","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-11T10:03:56Z","receivedAt":"2016-02-11T10:03:56Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> The refs infrastructure learns about log-only ref updates, which only\n> update the reflog. Later, we will use this to separate symbolic\n> reference resolution from ref updating.\n\nThis looks good. I assume it will get some testing later in the series.\n\n> [...]\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index fc5d1db..b5d0ab8 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -42,6 +42,8 @@\n> * value to ref_update::flags\n> */\n> \n> +#define REF_LOG_ONLY 0x80\n> +\n\nPlease add a comment explaining the meaning/purpose of this flag.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"277959","messageId":"1455225162.29013.10.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC4D94.4040003@alum.mit.edu","subject":"Re: [PATCH v4 09/21] refs: add method to rename refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-11T21:12:42Z","receivedAt":"2016-02-11T21:12:42Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 10:00 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Signed-off-by: David Turner \n> > ---\n> > refs.c | 5 +++++\n> > refs/files-backend.c | 4 +++-\n> > refs/refs-internal.h | 9 +++++++++\n> > 3 files changed, 17 insertions(+), 1 deletion(-)\n> > \n> > [...]\n> > diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> > index 5768fee..15fa99c 100644\n> > --- a/refs/refs-internal.h\n> > +++ b/refs/refs-internal.h\n> > @@ -67,6 +67,13 @@ int do_for_each_per_worktree_ref(const char\n> > *submodule, const char *base,\n> > \t\t\t\t each_ref_fn fn, int trim, int\n> > flags,\n> > \t\t\t\t void *cb_data);\n> > \n> > +/*\n> > + * Check if the new name does not conflict with any existing refs\n> > + * (other than possibly the old ref). Return 0 if the ref can be\n> > + * renamed to the new name.\n> > + */\n> > +int rename_ref_available(const char *oldname, const char\n> > *newname);\n> \n> Thanks for adding a docstring for this function. But this function\n> already has a declaration later in the file so you should remove one\n> or\n> the other.\n\nFixed, thanks.\n"},{"id":"277961","messageId":"1455225336.29013.11.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC4C3D.8040505@alum.mit.edu","subject":"Re: [PATCH v4 08/21] refs: add methods to init refs db","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-11T21:15:36Z","receivedAt":"2016-02-11T21:15:36Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 09:54 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Alternate refs backends might not need the refs/heads directory and\n> > so\n> > on, so we make ref db initialization part of the backend.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > builtin/init-db.c | 20 ++++++++++----------\n> > refs.c | 5 +++++\n> > refs.h | 2 ++\n> > refs/files-backend.c | 16 ++++++++++++++++\n> > refs/refs-internal.h | 2 ++\n> > 5 files changed, 35 insertions(+), 10 deletions(-)\n> > \n> > [...]\n> > diff --git a/refs.h b/refs.h\n> > index 5bc3fbc..c5ecc52 100644\n> > --- a/refs.h\n> > +++ b/refs.h\n> > @@ -66,6 +66,8 @@ extern int ref_exists(const char *refname);\n> > \n> > extern int is_branch(const char *refname);\n> > \n> > +extern int refs_init_db(struct strbuf *err, int shared);\n> \n> The convention elsewhere is for \"struct strbuf *err\" to be the *last*\n> parameter.\n\nFixed, thanks.\n"},{"id":"277962","messageId":"1455225682.29013.12.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC4AE4.6060708@alum.mit.edu","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-11T21:21:22Z","receivedAt":"2016-02-11T21:21:22Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 09:48 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Add a database backend for refs using LMDB. This backend runs git\n> > for-each-ref about 30% faster than the files backend with fully\n> > -packed\n> > refs on a repo with ~120k refs. It's also about 4x faster than\n> > using\n> > fully-unpacked refs. In addition, and perhaps more importantly, it\n> > avoids case-conflict issues on OS X.\n> \n> When I compile this using gcc with -Werror=pointer-arith, I get the\n> following errors:\n> \n> > refs/lmdb-backend.c: In function ‘rename_reflog_ent’:\n> > refs/lmdb-backend.c:1068:39: error: pointer of type ‘void *’ used\n> > in arithmetic [-Werror=pointer-arith]\n> > strbuf_add(&new_key_buf, key.mv_data + key.mv_size - 8, 8);\n> > ^\n> > refs/lmdb-backend.c:1068:53: error: pointer of type ‘void *’ used\n> > in arithmetic [-Werror=pointer-arith]\n> > strbuf_add(&new_key_buf, key.mv_data + key.mv_size - 8, 8);\n\nFixed, thanks.\n"},{"id":"277963","messageId":"1455225803.29013.13.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC5C8C.8060201@alum.mit.edu","subject":"Re: [PATCH v4 12/21] refs: allow log-only updates","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-11T21:23:23Z","receivedAt":"2016-02-11T21:23:23Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 11:03 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > The refs infrastructure learns about log-only ref updates, which\n> > only\n> > update the reflog. Later, we will use this to separate symbolic\n> > reference resolution from ref updating.\n> \n> This looks good. I assume it will get some testing later in the\n> series.\n\nThe existing tests cover this pretty well, I think. It's not intended\nto have user-visible results, so it's hard to specifically test. \n\n> > [...]\n> > diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> > index fc5d1db..b5d0ab8 100644\n> > --- a/refs/refs-internal.h\n> > +++ b/refs/refs-internal.h\n> > @@ -42,6 +42,8 @@\n> > * value to ref_update::flags\n> > */\n> > \n> > +#define REF_LOG_ONLY 0x80\n> > +\n> \n> Please add a comment explaining the meaning/purpose of this flag.\n\nCommented, thanks.\n"},{"id":"277984","messageId":"1455239298.29013.18.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC4985.5070707@alum.mit.edu","subject":"Re: [PATCH v4 03/21] refs: add methods for the ref iterators","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-12T01:08:18Z","receivedAt":"2016-02-12T01:08:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 09:42 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > From: Ronnie Sahlberg \n> > \n> > Signed-off-by: Ronnie Sahlberg \n> > Signed-off-by: David Turner \n> > ---\n> > refs.c | 54\n> > ++++++++++++++++++++++++++++++++++++++++++++++++++++\n> > refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n> > refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n> > 3 files changed, 112 insertions(+), 12 deletions(-)\n> > \n> > diff --git a/refs.c b/refs.c\n> > index afdde7d..e598b73 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -1158,3 +1158,57 @@ int resolve_gitlink_ref(const char *path,\n> > const char *refname,\n> > {\n> > \treturn the_refs_backend->resolve_gitlink_ref(path,\n> > refname, sha1);\n> > }\n> > +\n> > +int head_ref(each_ref_fn fn, void *cb_data)\n> > +{\n> > +\treturn the_refs_backend->head_ref(fn, cb_data);\n> > +}\n> > +\n> > +int head_ref_submodule(const char *submodule, each_ref_fn fn, void\n> > *cb_data)\n> > +{\n> > +\treturn the_refs_backend->head_ref_submodule(submodule, fn,\n> > cb_data);\n> > +}\n> > +\n> \n> I think it is unnecessary to have so many virtual functions. For\n> example, here you have made head_ref_submodule() virtual. But the\n> files\n> and lmdb implementations of this function are identical. They both\n> call\n> do_head_ref(), which are (confusingly) two independent static\n> functions.\n> But those functions are *also* defined identically, namely\n> \n> After all, what else would they possibly want to do? And both\n> resolve_gitlink_ref() and read_ref_full() are already virtual\n> functions\n> (actually read_ref_full() only has a single definition but it calls\n> the\n> virtualized resolve_ref_unsafe()).\n> \n> So it seems to me that it is unnecessary for head_ref_submodule() to\n> be\n> virtual, and that there only needs to be one definition of\n> do_head_ref().\n> \n> (Off-topic: for that matter, do_head_ref() is itself almost\n> pointless.\n> It could easily be inlined if we were sure that nobody passes it\n> submodule==NULL.)\n\nAdded a commit to move these to common code and inline do_head_ref.\n\n> Similarly, I bet that if do_for_each_ref() were just a little bit\n> more\n> capable, then the following functions could also remain non-virtual\n> (i.e., one definition could be shared across all backend\n> implementations):\n> \n> * for_each_ref()\n> ...\n\nYeah, that makes sense. I think we've grown a few more of those funcs\nsince Ronnie's version, so virtualizing do_for_each_ref makes more\nsense.\n"},{"id":"277985","messageId":"1455239343.29013.19.camel@twopensource.com","threadId":"41344","inReplyTo":"56BC3C08.2080003@alum.mit.edu","subject":"Re: [PATCH v4 02/21] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-12T01:09:03Z","receivedAt":"2016-02-12T01:09:03Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-11 at 08:45 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > From: Ronnie Sahlberg \n> > \n> > Add ref backend methods for:\n> > resolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,\n> > create_symref, resolve_gitlink_ref.\n> > \n> > Signed-off-by: Ronnie Sahlberg \n> > Signed-off-by: David Turner \n> > ---\n> > builtin/init-db.c | 1 +\n> > refs.c | 36 ++++++++++++++++++++++++++++++++++++\n> > refs/files-backend.c | 33 +++++++++++++++++++++++----------\n> > refs/refs-internal.h | 23 +++++++++++++++++++++++\n> > 4 files changed, 83 insertions(+), 10 deletions(-)\n> > \n> > diff --git a/builtin/init-db.c b/builtin/init-db.c\n> > index 07229d6..26e1cc3 100644\n> > --- a/builtin/init-db.c\n> > +++ b/builtin/init-db.c\n> > @@ -8,6 +8,7 @@\n> > #include \"builtin.h\"\n> > #include \"exec_cmd.h\"\n> > #include \"parse-options.h\"\n> > +#include \"refs.h\"\n> \n> You can't see it in this diff's context, but there is already an\n> '#include \"refs.h\"' in this file.\n\nFixed this.\n\nI think that's all of your comments so far, but I'm going to wait for\nyour second batch to reroll, to avoid unnecessary churn.\n"},{"id":"278009","messageId":"56BDE780.2030407@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-14-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 13/21] refs: resolve symbolic refs first","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-12T14:09:04Z","receivedAt":"2016-02-12T14:09:04Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Before committing ref updates, split symbolic ref updates into two\n> parts: an update to the underlying ref, and a log-only update to the\n> symbolic ref. This ensures that both references are locked correctly\n> while their reflogs are updated.\n> \n> It is still possible to confuse git by concurrent updates, since the\n> splitting of symbolic refs does not happen under lock. So a symbolic ref\n> could be replaced by a plain ref in the middle of this operation, which\n> would lead to reflog discontinuities and missed old-ref checks.\n\nThis patch is doing too much at once for my little brain to follow.\n\nMy first hangup is the change to setting RESOLVE_REF_NO_RECURSE\nunconditionally in lock_ref_sha1_basic(). I count five callers of that\nfunction and see no justification for why the change is OK in the\ncontext of each caller. Here are some thoughts:\n\n* The call from files_create_symref() sets REF_NODEREF, so it is\nunaffected by this change.\n\n* The call from files_transaction_commit() is preceded by a call to\ndereference_symrefs(), which I assume effectively replaces the need for\nRESOLVE_REF_NO_RECURSE.\n\n* There are two calls from files_rename_ref(). Why is it OK to do\nwithout RESOLVE_REF_NO_RECURSE there?\n\n * For the oldrefname call, I suppose the justification is the \"(flag &\nREF_ISSYMREF)\" check earlier in the function. (But does this introduce a\nsignificant TOCTOU race?)\n\n * For the newrefname call, I suppose it's because the code a little\nhigher up tries to delete any existing reference with that name. It\nlooks to me like the old code was slightly broken: if newrefname was an\nunborn symbolic reference, then: read_ref_full() would fail;\ndelete_ref() would be skipped; lock_ref_sha1_basic() would lock the\n*referred-to* reference; the referred-to reference would be overwritten\ninstead of newrefname. So it could be that here REF_NODEREF indirectly\nfixes a bug?\n\n* The last call, from files_reflog_expire(), is also questionable before\nyour patch. If refname is a symref, then the function is expiring the\nreflog of the symref. But (before this patch) it locks not the symref\nbut its referent. This was discussed in some length before on the\nmailing list [1], and the conclusion was that the current behavior is\nwrong, but for backwards compatibility reasons it would be safest to\nchange it to locking *both* the symref and its referent.\n\nIf possible, it would be better to split this patch up into several: the\nfirst few would each add the REF_NODEREF flag at one callsite, with a\ncareful justification of why that is OK. Once all the callsites (except\nthe one in files_transaction_commit()) have been changed, then the last\npatch could add the dereference_symrefs() machinery and change the last\ncallsite.\n\n(I'm not certain that those steps are actually doable independently,\ngiven that REF_NODEREF has other effects besides setting\nRESOLVE_REF_NO_RECURSE.)\n\nI'm not just being pedantic here. The patch as written is really too big\nto review effectively.\n\nMichael\n\n[1]\nhttp://thread.gmane.org/gmane.comp.version-control.git/263552/focus=263555\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278010","messageId":"56BDF531.7010909@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-15-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 14/21] refs: always handle non-normal refs in files backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-12T15:07:29Z","receivedAt":"2016-02-12T15:07:29Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Always handle non-normal (per-worktree or pseudo) refs in the files\n> backend instead of alternate backends.\n> \n> Sometimes a ref transaction will update both a per-worktree ref and a\n> normal ref. For instance, an ordinary commit might update\n> refs/heads/master and HEAD (or at least HEAD's reflog).\n> \n> Updates to normal refs continue to go through the chosen backend.\n> \n> Updates to non-normal refs are moved to a separate files backend\n> transaction.\n> \n> Signed-off-by: David Turner \n> ---\n> refs.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n> 1 file changed, 79 insertions(+), 2 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 227c018..18ba356 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -9,6 +9,11 @@\n> #include \"object.h\"\n> #include \"tag.h\"\n> \n> +static const char split_transaction_fail_warning[] = N_(\n> +\t\"A ref transaction was split across two refs backends. Part of the \"\n> +\t\"transaction succeeded, but then the update to the per-worktree refs \"\n> +\t\"failed. Your repository may be in an inconsistent state.\");\n> +\n> /*\n> * We always have a files backend and it is the default.\n> */\n> @@ -791,6 +796,13 @@ void ref_transaction_free(struct ref_transaction *transaction)\n> \tfree(transaction);\n> }\n> \n> +static void add_update_obj(struct ref_transaction *transaction,\n> +\t\t\t struct ref_update *update)\n> +{\n> +\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n> +\ttransaction->updates[transaction->nr++] = update;\n> +}\n> +\n> static struct ref_update *add_update(struct ref_transaction *transaction,\n> \t\t\t\t const char *refname)\n> {\n> @@ -798,8 +810,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n> \tstruct ref_update *update = xcalloc(1, sizeof(*update) + len);\n> \n> \tmemcpy((char *)update->refname, refname, len); /* includes NUL */\n> -\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n> -\ttransaction->updates[transaction->nr++] = update;\n> +\tadd_update_obj(transaction, update);\n> \treturn update;\n> }\n> \n> @@ -1217,11 +1228,38 @@ static int dereference_symrefs(struct ref_transaction *transaction,\n> \treturn 0;\n> }\n> \n> +/*\n> + * Move all non-normal ref updates into a specially-created\n> + * files-backend transaction\n> + */\n> +static int move_abnormal_ref_updates(struct ref_transaction *transaction,\n> +\t\t\t\t struct ref_transaction *files_transaction,\n> +\t\t\t\t struct strbuf *err)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\tint last;\n> +\t\tstruct ref_update *update = transaction->updates[i];\n> +\n> +\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n> +\t\t\tcontinue;\n> +\n> +\t\tlast = --transaction->nr;\n> +\t\ttransaction->updates[i] = transaction->updates[last];\n> +\t\tadd_update_obj(files_transaction, update);\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n\nI think this function is incorrect. The update that was previously at\ntransaction->updates[i] never gets checked for abnormality. You could\nfix that and also avoid gratuitously changing the order of the updates\nlike this:\n\nstatic int move_abnormal_ref_updates(struct ref_transaction *transaction,\n\t\t\t\t struct ref_transaction *files_transaction,\n\t\t\t\t struct strbuf *err)\n{\n\tint i, normal_nr = 0;\n\n\tfor (i = 0; i < transaction->nr; i++) {\n\t\tstruct ref_update *update = transaction->updates[i];\n\n\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n\t\t\ttransaction->updates[normal_nr++] = update;\n\t\telse\n\t\t\tadd_update_obj(files_transaction, update);\n\t}\n\n\ttransaction->nr = normal_nr;\n\treturn 0;\n}\n\nAnother alternative would be to set\n\n update->flags |= REF_ABNORMAL\n\nfor the abnormal references and *leave them* in the original transaction\nwhile also adding them to files_transactions. Then teach non-files\nbackends to skip over updates with REF_ABNORMAL.\n\nThe reason I thought of this is that long-term I'd like to make it\npossible for some reference updates to fail without aborting the\ntransaction. To implement that, we would want a way for\nref_transaction_commit() to report back to its caller *which* updates\nfailed, and an obvious way to do that would be to store the\nreference-specific errors in struct ref_update. If you leave the\nabnormal ref_updates in the main transaction, then my hoped-for change\nwould be easier.\n\nBut that's a separate and hypothetical idea, so you don't have to let it\ninfluence how you implement this patch.\n\n> int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t struct strbuf *err)\n> {\n> \tint ret = -1;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct ref_transaction *files_transaction = NULL;\n> \n> \tassert(err);\n> \n> @@ -1237,6 +1275,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \tif (ret)\n> \t\tgoto done;\n> \n> +\tif (the_refs_backend != &refs_be_files) {\n> +\t\tfiles_transaction = ref_transaction_begin(err);\n> +\t\tif (!files_transaction)\n> +\t\t\tgoto done;\n> +\n> +\t\tret = move_abnormal_ref_updates(transaction, files_transaction,\n> +\t\t\t\t\t\terr);\n> +\t\tif (ret)\n> +\t\t\tgoto done;\n> +\n> +\t\t/* files backend commit */\n> +\t\tif (get_affected_refnames(files_transaction,\n> +\t\t\t\t\t\t &files_affected_refnames,\n> +\t\t\t\t\t\t err)) {\n> +\t\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\t\tgoto done;\n> +\t\t}\n> +\t}\n> +\n> +\t/* main backend commit */\n> \tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n> \t\tret = TRANSACTION_GENERIC_ERROR;\n> \t\tgoto done;\n> @@ -1244,8 +1302,24 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \n> \tret = the_refs_backend->transaction_commit(transaction,\n> \t\t\t\t\t\t &affected_refnames, err);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tif (files_transaction) {\n> +\t\tret = refs_be_files.transaction_commit(files_transaction,\n> +\t\t\t\t\t\t &files_affected_refnames,\n> +\t\t\t\t\t\t err);\n> +\t\tif (ret) {\n> +\t\t\twarning(split_transaction_fail_warning);\n> +\t\t\tgoto done;\n> +\t\t}\n> +\t}\n> +\n> done:\n> \tstring_list_clear(&affected_refnames, 0);\n> +\tstring_list_clear(&files_affected_refnames, 0);\n> +\tif (files_transaction)\n> +\t\tref_transaction_free(files_transaction);\n> \treturn ret;\n> }\n> \n> @@ -1285,6 +1359,9 @@ int peel_ref(const char *refname, unsigned char *sha1)\n> int create_symref(const char *ref_target, const char *refs_heads_master,\n> \t\t const char *logmsg)\n> {\n> +\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n> +\t\treturn refs_be_files.create_symref(ref_target, refs_heads_master,\n> +\t\t\t\t\t\t logmsg);\n> \treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n> \t\t\t\t\t logmsg);\n> }\n> \n\nDoes initial_ref_transaction_commit() need the same treatment?\n\nI think files_rename_ref() will break if one of the references is normal\nand one is abnormal. I think it would be OK to prohibit renaming\nabnormal refs entirely (can anybody think of an important use case?),\nbut that function should at least do the checks.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278012","messageId":"56BDF9B1.5050302@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-16-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 15/21] init: allow alternate ref strorage to be set for new repos","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-12T15:26:41Z","receivedAt":"2016-02-12T15:26:41Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> git init learns a new argument --ref-storage. Presently, only\n> \"files\" is supported, but later we will add other storage backends.\n> \n> When this argument is used, the repository's extensions.refStorage\n> configuration value is set (as well as core.repositoryformatversion),\n> and the ref storage backend's initdb function is used to set up the\n> ref database.\n> \n> Signed-off-by: David Turner \n> Signed-off-by: SZEDER Gábor \n> ---\n> Documentation/git-init-db.txt | 2 +-\n> Documentation/git-init.txt | 7 +++++-\n> builtin/init-db.c | 44 +++++++++++++++++++++++++++-------\n> cache.h | 2 ++\n> contrib/completion/git-completion.bash | 3 ++-\n> path.c | 29 ++++++++++++++++++++--\n> refs.c | 8 +++++++\n> refs.h | 8 +++++++\n> setup.c | 12 ++++++++++\n> t/t0001-init.sh | 25 +++++++++++++++++++\n> 10 files changed, 127 insertions(+), 13 deletions(-)\n> \n> [...]\n> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n> index 8174d27..d2b150f 100644\n> --- a/Documentation/git-init.txt\n> +++ b/Documentation/git-init.txt\n> @@ -12,7 +12,7 @@ SYNOPSIS\n> 'git init' [-q | --quiet] [--bare] [--template=]\n> \t [--separate-git-dir ]\n> \t [--shared[=]] [directory]\n> -\n> +\t [--ref-storage=]\n> \n> DESCRIPTION\n> -----------\n> @@ -113,6 +113,11 @@ does not exist, it will be created.\n> \n> --\n> \n> +--ref-storage=::\n> +Type of refs storage backend. Default is to use the original \"files\"\n> +storage, which stores ref data in files in .git/refs and\n> +.git/packed-refs.\n> +\n\nTechnically, that should be $GIT_DIR/refs and $GIT_DIR/packed-refs. But\nit might be that we are not so picky about the distinction in the user docs.\n\n> TEMPLATE DIRECTORY\n> ------------------\n> \n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 801e977..d331ce8 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -24,6 +24,7 @@ static int init_is_bare_repository = 0;\n> static int init_shared_repository = -1;\n> static const char *init_db_template_dir;\n> static const char *git_link;\n> +static char *requested_ref_storage_backend;\n> \n> static void copy_templates_1(struct strbuf *path, struct strbuf *template,\n> \t\t\t DIR *dir)\n> @@ -179,6 +180,7 @@ static int create_default_files(const char *template_path)\n> \tint reinit;\n> \tint filemode;\n> \tstruct strbuf err = STRBUF_INIT;\n> +\tint repo_version = 0;\n> \n> \t/* Just look for `init.templatedir` */\n> \tgit_config(git_init_db_config, NULL);\n> @@ -205,6 +207,32 @@ static int create_default_files(const char *template_path)\n> \t}\n> \n> \t/*\n> +\t * Create the default symlink from \".git/HEAD\" to the \"master\"\n\nI know that you are just moving this comment, but s/symlink/symref/\nwould make it more up-to-date.\n\n> +\t * branch, if it does not exist yet.\n> +\t */\n> [...]\n> diff --git a/setup.c b/setup.c\n> index 0deb022..1a62277 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1,5 +1,6 @@\n> #include \"cache.h\"\n> #include \"dir.h\"\n> +#include \"refs.h\"\n> #include \"string-list.h\"\n> \n> static int inside_git_dir = -1;\n> @@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n> \treturn ret;\n> }\n> \n> +int ref_storage_backend_config(const char *var, const char *value, void *ptr)\n> +{\n> +\tchar **cdata = ptr;\n> +\n> +\tif (!strcmp(var, \"extensions.refstorage\"))\n> +\t\t*cdata = xstrdup(value);\n> +\treturn 0;\n> +}\n> +\n> /*\n> * Test if it looks like we're at a git directory.\n> * We want to see:\n> @@ -390,6 +400,8 @@ static int check_repo_format(const char *var, const char *value, void *cb)\n> \t\t\t;\n> \t\telse if (!strcmp(ext, \"preciousobjects\"))\n> \t\t\trepository_format_precious_objects = git_config_bool(var, value);\n> +\t\telse if (!strcmp(ext, \"refstorage\"))\n> +\t\t\tref_storage_backend = xstrdup(value);\n> \t\telse\n> \t\t\tstring_list_append(&unknown_extensions, ext);\n> \t}\n> diff --git a/t/t0001-init.sh b/t/t0001-init.sh\n> index 295aa59..31c8279 100755\n> --- a/t/t0001-init.sh\n> +++ b/t/t0001-init.sh\n> @@ -174,6 +174,31 @@ test_expect_success 'reinit' '\n> \ttest_i18ncmp again/empty again/err2\n> '\n> \n> +test_expect_success 'init with bogus storage backend fails' '\n> +\n> +\t(\n> +\t\tmkdir again2 &&\n> +\t\tcd again2 &&\n> +\t\ttest_must_fail git init --ref-storage=test >out2 2>err2 &&\n> +\t\ttest_i18ngrep \"Unknown ref storage backend test\" err2\n> +\t)\n> +'\n> +\n> +test_expect_failure 'reinit with changed storage backend fails' '\n> +\n> +\t(\n> +\t\tmkdir again3 &&\n> +\t\tcd again3 &&\n> +\t\tgit init >out1 2>err1 &&\n> +\t\tgit init --ref-storage=test >out2 2>err2\n> +\t) &&\n> +\ttest_i18ngrep \"Initialized empty\" again3/out1 &&\n> +\ttest_i18ngrep \"Reinitialized existing\" again3/out2 &&\n> +\t>again3/empty &&\n> +\ttest_i18ncmp again3/empty again3/err1 &&\n> +\ttest_i18ncmp again3/empty again3/err2\n> +'\n> +\n\nIs it worth testing re-initializing with the same --ref-storage? Perhaps\nnot.\n\n> test_expect_success 'init with --template' '\n> \tmkdir template-source &&\n> \techo content >template-source/file &&\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278013","messageId":"56BDFD4C.3050809@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-20-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 19/21] refs: add register_ref_storage_backends()","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-12T15:42:04Z","receivedAt":"2016-02-12T15:42:04Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> This new function will register all known ref storage backends... once\n> there are any other than the default. For now, it's a no-op.\n> \n> Signed-off-by: David Turner \n> ---\n> builtin/init-db.c | 3 +++\n> config.c | 25 +++++++++++++++++++++++++\n> refs.c | 8 ++++++++\n> refs.h | 2 ++\n> 4 files changed, 38 insertions(+)\n> \n> [...]\n> diff --git a/config.c b/config.c\n> index b95ac3a..b9ef223 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -11,6 +11,7 @@\n> #include \"strbuf.h\"\n> #include \"quote.h\"\n> #include \"hashmap.h\"\n> +#include \"refs.h\"\n> #include \"string-list.h\"\n> #include \"utf8.h\"\n> \n> @@ -1207,6 +1208,30 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n> \t}\n> \n> \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n> +\t\tchar *storage = NULL;\n> +\n> +\t\t/*\n> +\t\t * make sure we always read the ref storage config\n> +\t\t * from the extensions section on startup\n> +\t\t */\n> +\t\tret += git_config_from_file(ref_storage_backend_config,\n> +\t\t\t\t\t repo_config, &storage);\n> +\n> +\t\tregister_ref_storage_backends();\n> +\t\tif (!storage)\n> +\t\t\tstorage = xstrdup(\"\");\n> +\n> +\t\tif ((!*storage) ||\n> +\t\t (!strcmp(storage, \"files\"))) {\n\nNit: you have some unnecessary parentheses here.\n\n> +\t\t\t/* default backend, nothing to do */\n> +\t\t\tfree(storage);\n> +\t\t} else {\n> +\t\t\tref_storage_backend = storage;\n> +\t\t\tif (set_ref_storage_backend(ref_storage_backend))\n> +\t\t\t\tdie(_(\"Unknown ref storage backend %s\"),\n> +\t\t\t\t ref_storage_backend);\n> +\t\t}\n> +\n> \t\tret += git_config_from_file(fn, repo_config, data);\n> \t\tfound += 1;\n> \t}\n> diff --git a/refs.c b/refs.c\n> index 715a492..e50cca0 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1554,3 +1554,11 @@ done:\n> \tstring_list_clear(&affected_refnames, 0);\n> \treturn ret;\n> }\n> +\n> +void register_ref_storage_backends(void) {\n> +\t/*\n> +\t * No need to register the files backend; it's registered by\n> +\t * default. Add register_ref_storage_backend(ptr-to-backend)\n> +\t * entries below when you add a new backend.\n> +\t */\n\nThis function must be called every run, right? So why not make it\nregister the \"files\" backend explicitly? That would make it obvious\nreally quick if this function fails to get called in some code path.\nJust a thought...\n\n> +}\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278017","messageId":"56BE0FE1.5030407@alum.mit.edu","threadId":"41344","inReplyTo":"1454701462-3817-21-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-12T17:01:21Z","receivedAt":"2016-02-12T17:01:21Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/05/2016 08:44 PM, David Turner wrote:\n> Add a database backend for refs using LMDB. This backend runs git\n> for-each-ref about 30% faster than the files backend with fully-packed\n> refs on a repo with ~120k refs. It's also about 4x faster than using\n> fully-unpacked refs. In addition, and perhaps more importantly, it\n> avoids case-conflict issues on OS X.\n> \n> LMDB has a few features that make it suitable for usage in git:\n> \n\n0. It is licensed under the OpenLDAP Public License, which is apparently\nGPL compatible [1].\n\n[1] http://www.gnu.org/licenses/license-list.en.html#newOpenLDAP\n\n> 1. It is relatively lightweight; it requires only one header file, and\n> the library code takes under 64k at runtime.\n> \n> 2. It is well-tested: it's been used in OpenLDAP for years.\n> \n> 3. It's very fast. LMDB's benchmarks show that it is among\n> the fastest key-value stores.\n> \n> 4. It has a relatively simple concurrency story; readers don't\n> block writers and writers don't block readers.\n> \n> Ronnie Sahlberg's original version of this patchset used tdb. The\n> major disadvantage of tdb is that tdb is hard to build on OS X. It's\n> also not in homebrew. So lmdb seemed simpler.\n> \n> To test this backend's correctness, I hacked test-lib.sh and\n> test-lib-functions.sh to run all tests under the refs backend. Dozens\n> of tests use manual ref/reflog reading/writing, or create submodules\n> without passing --ref-storage to git init. If those tests are\n> changed to use the update-ref machinery or test-refs-lmdb-backend (or,\n> in the case of packed-refs, corrupt refs, and dumb fetch tests, are\n> skipped), the only remaining failing tests are the git-new-workdir\n> tests and the gitweb tests.\n\nWould it possible to build your \"hack\" into the test suite? For example,\nperhaps one could implement an option that requests tests to use LMDB\nwherever possible and skip the tests that are not LMDB-compatible. Over\ntime, we could try to increase the fraction of tests that are\nLMDB-compatible by (for example) using `git update-ref` and `git\nrev-parse` rather than reading/writing reference files via the\nfilesystem directly.\n\nOtherwise, I'm worried that LMDB support will bitrot quickly because it\nwill be too much of a nuisance to exercise the code.\n\n> Signed-off-by: David Turner \n> ---\n> .gitignore | 1 +\n> Documentation/config.txt | 7 +\n> Documentation/git-clone.txt | 3 +-\n> Documentation/git-init.txt | 2 +-\n> Documentation/technical/refs-lmdb-backend.txt | 52 +\n> Documentation/technical/repository-version.txt | 5 +\n> Makefile | 12 +\n> configure.ac | 33 +\n> contrib/workdir/git-new-workdir | 3 +\n> refs.c | 3 +\n> refs.h | 2 +\n> refs/lmdb-backend.c | 2052 ++++++++++++++++++++++++\n> setup.c | 11 +-\n> test-refs-lmdb-backend.c | 64 +\n> transport.c | 7 +-\n> 15 files changed, 2250 insertions(+), 7 deletions(-)\n> create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n> create mode 100644 refs/lmdb-backend.c\n> create mode 100644 test-refs-lmdb-backend.c\n> \n> diff --git a/.gitignore b/.gitignore\n> index 1c2f832..87d45a2 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -199,6 +199,7 @@\n> /test-path-utils\n> /test-prio-queue\n> /test-read-cache\n> +/test-refs-lmdb-backend\n> /test-regex\n> /test-revision-walking\n> /test-run-command\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 06d3659..bc222c9 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1153,6 +1153,13 @@ difftool..cmd::\n> difftool.prompt::\n> \tPrompt before each invocation of the diff tool.\n> \n> +extensions.refsStorage::\n> +\tType of refs storage backend. Default is to use the original\n> +\tfiles based ref storage. When set to \"lmdb\", refs are stored in\n> +\tthe lmdb database backend. This setting reflects the refs\n> +\tbackend that is currently in use; it is not possible to change\n> +\tthe backend by changing this setting.\n> +\n\nLet's give the users a pointer to how they *can* change this setting. Maybe\n\n\tType of refs storage backend. Default is to use the original\n\tfiles based ref storage. When set to \"lmdb\", refs are stored in\n\tan LMDB database. This setting reflects the refs storage\n\tbackend that was chosen via the --ref-storage option when the\n\trepository was originally created. It is currently\n\tnot possible to change the refs storage backend of an\n\texisting repository.\n\n\n> fetch.recurseSubmodules::\n> \tThis option can be either set to a boolean value or to 'on-demand'.\n> \tSetting it to a boolean changes the behavior of fetch and pull to\n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 68f56a7..b9c52ce 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -227,7 +227,8 @@ objects from the source repository into a pack in the cloned repository.\n> \n> --ref-storage=::\n> \tType of ref storage backend. Default is to use the original files\n> -\tbased ref storage.\n> +\tbased ref storage. Set to \"lmdb\" to store refs in the lmdb database\n> +\tbackend.\n\nShould the second \"lmdb\" be upper-case? Maybe also in other places where\nthe LMDB database is being referred to?\n\n> \n> ::\n> \tThe (possibly remote) repository to clone from. See the\n> [...]\n\nI still haven't reviewed the actual lmdb-backend code. I guess I'm going\nto have to make myself an LMDB expert one of these days...\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278061","messageId":"1455326623.10888.3.camel@twopensource.com","threadId":"41344","inReplyTo":"56BE0FE1.5030407@alum.mit.edu","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-13T01:23:43Z","receivedAt":"2016-02-13T01:23:43Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-02-12 at 18:01 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Add a database backend for refs using LMDB. This backend runs git\n> > for-each-ref about 30% faster than the files backend with fully\n> > -packed\n> > refs on a repo with ~120k refs. It's also about 4x faster than\n> > using\n> > fully-unpacked refs. In addition, and perhaps more importantly, it\n> > avoids case-conflict issues on OS X.\n> > \n> > LMDB has a few features that make it suitable for usage in git:\n> > \n> \n> 0. It is licensed under the OpenLDAP Public License, which is\n> apparently\n> GPL compatible [1].\n> \n> [1] http://www.gnu.org/licenses/license-list.en.html#newOpenLDAP\n\nOh, yeah, I checked that before choosing it and then forgot all about\nit. \n\n> > 1. It is relatively lightweight; it requires only one header file,\n> > and\n> > the library code takes under 64k at runtime.\n> > \n> > 2. It is well-tested: it's been used in OpenLDAP for years.\n> > \n> > 3. It's very fast. LMDB's benchmarks show that it is among\n> > the fastest key-value stores.\n> > \n> > 4. It has a relatively simple concurrency story; readers don't\n> > block writers and writers don't block readers.\n> > \n> > Ronnie Sahlberg's original version of this patchset used tdb. The\n> > major disadvantage of tdb is that tdb is hard to build on OS X. \n> > It's\n> > also not in homebrew. So lmdb seemed simpler.\n> > \n> > To test this backend's correctness, I hacked test-lib.sh and\n> > test-lib-functions.sh to run all tests under the refs backend.\n> > Dozens\n> > of tests use manual ref/reflog reading/writing, or create\n> > submodules\n> > without passing --ref-storage to git init. If those tests are\n> > changed to use the update-ref machinery or test-refs-lmdb-backend\n> > (or,\n> > in the case of packed-refs, corrupt refs, and dumb fetch tests, are\n> > skipped), the only remaining failing tests are the git-new-workdir\n> > tests and the gitweb tests.\n> \n> Would it possible to build your \"hack\" into the test suite? For\n> example,\n> perhaps one could implement an option that requests tests to use LMDB\n> wherever possible and skip the tests that are not LMDB-compatible.\n> Over\n> time, we could try to increase the fraction of tests that are\n> LMDB-compatible by (for example) using `git update-ref` and `git\n> rev-parse` rather than reading/writing reference files via the\n> filesystem directly.\n> \n> Otherwise, I'm worried that LMDB support will bitrot quickly because\n> it\n> will be too much of a nuisance to exercise the code.\n\nI can add that. It requires minor changes to a large number of tests,\nbut that's OK.\n\n> > Signed-off-by: David Turner \n> > ---\n> > .gitignore | 1 +\n> > Documentation/config.txt | 7 +\n> > Documentation/git-clone.txt | 3 +-\n> > Documentation/git-init.txt | 2 +-\n> > Documentation/technical/refs-lmdb-backend.txt | 52 +\n> > Documentation/technical/repository-version.txt | 5 +\n> > Makefile | 12 +\n> > configure.ac | 33 +\n> > contrib/workdir/git-new-workdir | 3 +\n> > refs.c | 3 +\n> > refs.h | 2 +\n> > refs/lmdb-backend.c | 2052\n> > ++++++++++++++++++++++++\n> > setup.c | 11 +-\n> > test-refs-lmdb-backend.c | 64 +\n> > transport.c | 7 +-\n> > 15 files changed, 2250 insertions(+), 7 deletions(-)\n> > create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n> > create mode 100644 refs/lmdb-backend.c\n> > create mode 100644 test-refs-lmdb-backend.c\n> > \n> > diff --git a/.gitignore b/.gitignore\n> > index 1c2f832..87d45a2 100644\n> > --- a/.gitignore\n> > +++ b/.gitignore\n> > @@ -199,6 +199,7 @@\n> > /test-path-utils\n> > /test-prio-queue\n> > /test-read-cache\n> > +/test-refs-lmdb-backend\n> > /test-regex\n> > /test-revision-walking\n> > /test-run-command\n> > diff --git a/Documentation/config.txt b/Documentation/config.txt\n> > index 06d3659..bc222c9 100644\n> > --- a/Documentation/config.txt\n> > +++ b/Documentation/config.txt\n> > @@ -1153,6 +1153,13 @@ difftool..cmd::\n> > difftool.prompt::\n> > \tPrompt before each invocation of the diff tool.\n> > \n> > +extensions.refsStorage::\n> > +\tType of refs storage backend. Default is to use the\n> > original\n> > +\tfiles based ref storage. When set to \"lmdb\", refs are\n> > stored in\n> > +\tthe lmdb database backend. This setting reflects the refs\n> > +\tbackend that is currently in use; it is not possible to\n> > change\n> > +\tthe backend by changing this setting.\n> > +\n> \n> Let's give the users a pointer to how they *can* change this setting.\n> Maybe\n> \n> \tType of refs storage backend. Default is to use the original\n> \tfiles based ref storage. When set to \"lmdb\", refs are stored\n> in\n> \tan LMDB database. This setting reflects the refs storage\n> \tbackend that was chosen via the --ref-storage option when the\n> \trepository was originally created. It is currently\n> \tnot possible to change the refs storage backend of an\n> \texisting repository.\n\nOK.\n\n> > fetch.recurseSubmodules::\n> > \tThis option can be either set to a boolean value or to 'on\n> > -demand'.\n> > \tSetting it to a boolean changes the behavior of fetch and\n> > pull to\n> > diff --git a/Documentation/git-clone.txt b/Documentation/git\n> > -clone.txt\n> > index 68f56a7..b9c52ce 100644\n> > --- a/Documentation/git-clone.txt\n> > +++ b/Documentation/git-clone.txt\n> > @@ -227,7 +227,8 @@ objects from the source repository into a pack\n> > in the cloned repository.\n> > \n> > --ref-storage=::\n> > \tType of ref storage backend. Default is to use the\n> > original files\n> > -\tbased ref storage.\n> > +\tbased ref storage. Set to \"lmdb\" to store refs in the lmdb\n> > database\n> > +\tbackend.\n> \n> Should the second \"lmdb\" be upper-case? Maybe also in other places\n> where\n> the LMDB database is being referred to?\n\nOK, I think I got most of them.\n\n> > ::\n> > \tThe (possibly remote) repository to clone from. See the\n> > [...]\n> \n> I still haven't reviewed the actual lmdb-backend code. I guess I'm\n> going\n> to have to make myself an LMDB expert one of these days...\n\nI highly recommend it! The API is pretty reasonable, and it does seem\nto be fast. The error messages could use some work, but that's my only\ncomplaint.\n"},{"id":"278106","messageId":"CACsJy8CWS9E0Jb4omm4k0=kZ73w9UoHmofLFCFHUhHu53zfgYw@mail.gmail.com","threadId":"41344","inReplyTo":"1454701462-3817-21-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-02-14T12:04:16Z","receivedAt":"2016-02-14T12:04:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Feb 6, 2016 at 2:44 AM, David Turner wrote:\n> +static char *get_refdb_path(const char *base)\n> +{\n> + static struct strbuf path_buf = STRBUF_INIT;\n> + strbuf_reset(&path_buf);\n> + strbuf_addf(&path_buf, \"%s/refdb\", base);\n> + return path_buf.buf;\n> +}\n...\n> +static int lmdb_init_db(struct strbuf *err, int shared)\n> +{\n> + /*\n> + * To create a db, all we need to do is make a directory for\n> + * it to live in; lmdb will do the rest.\n> + */\n> +\n> + if (!db_path)\n> + db_path = xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n\nThis works for multiple worktrees. But scripts may have harder time\ngetting the path. The recommended way is \"git rev-parse --git-path\nrefdb\" but because \"refdb\" is not registered in path.c:common_list[],\nthat command becomes git_path(\"refdb\") instead of\nget_refdb(get_git_... like here. And I will need to know that\n.git/refdb is _not_ per-worktree when I migrate/convert main worktree\n(it's very likely I have to go that route to solve .git/config issue\nin multi worktree).\n\nThe solution is register refdb to common_list[] and you can do\ngit_path(\"refdb\") here. But then what happens when another backend is\nadded? Will the new backend use the same path \"refdb\", or say\n\"refdb.sqlite\"? If all backends share the name \"refdb\", why can't we\njust reuse \"refs\" instead because the default filesystem-based backend\nis technically just another backend?\n-- \nDuy\n"},{"id":"278147","messageId":"CACsJy8AkWpVERj8=ABaQ89xe=6-DRLV5KBz9_sxW8TE4LUzcQw@mail.gmail.com","threadId":"41344","inReplyTo":"CACsJy8CWS9E0Jb4omm4k0=kZ73w9UoHmofLFCFHUhHu53zfgYw@mail.gmail.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-02-15T09:57:09Z","receivedAt":"2016-02-15T09:57:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Feb 14, 2016 at 7:04 PM, Duy Nguyen wrote:\n> On Sat, Feb 6, 2016 at 2:44 AM, David Turner wrote:\n>> +static char *get_refdb_path(const char *base)\n>> +{\n>> + static struct strbuf path_buf = STRBUF_INIT;\n>> + strbuf_reset(&path_buf);\n>> + strbuf_addf(&path_buf, \"%s/refdb\", base);\n>> + return path_buf.buf;\n>> +}\n> ...\n>> +static int lmdb_init_db(struct strbuf *err, int shared)\n>> +{\n>> + /*\n>> + * To create a db, all we need to do is make a directory for\n>> + * it to live in; lmdb will do the rest.\n>> + */\n>> +\n>> + if (!db_path)\n>> + db_path = xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n>\n> This works for multiple worktrees. But scripts may have harder time\n> getting the path. The recommended way is \"git rev-parse --git-path\n> refdb\" but because \"refdb\" is not registered in path.c:common_list[],\n> that command becomes git_path(\"refdb\") instead of\n> get_refdb(get_git_... like here. And I will need to know that\n> .git/refdb is _not_ per-worktree when I migrate/convert main worktree\n> (it's very likely I have to go that route to solve .git/config issue\n> in multi worktree).\n>\n> The solution is register refdb to common_list[] and you can do\n> git_path(\"refdb\") here. But then what happens when another backend is\n> added? Will the new backend use the same path \"refdb\", or say\n> \"refdb.sqlite\"? If all backends share the name \"refdb\", why can't we\n> just reuse \"refs\" instead because the default filesystem-based backend\n> is technically just another backend?\n\nTo answer myself: I forgot that there were per-worktree refs (e.g.\nrefs/bisect). It makes me wonder if we should put per-worktree refs to\nlmdb as well (maybe one per worktree if we don't want to put all in\none db). One of the advantages of moving away from fs-based backend is\nthe ability to deal with case sensitivity, \"nested\" refs (e.g. a/b and\na/b/c are both refs). With this split, I think some refs are still\nleft behind.. Sorry if this was discussed before, I haven't followed\nthis closely.\n-- \nDuy\n"},{"id":"278395","messageId":"1455660081.7528.8.camel@twopensource.com","threadId":"41344","inReplyTo":"CACsJy8AkWpVERj8=ABaQ89xe=6-DRLV5KBz9_sxW8TE4LUzcQw@mail.gmail.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-16T22:01:21Z","receivedAt":"2016-02-16T22:01:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2016-02-15 at 16:57 +0700, Duy Nguyen wrote:\n> On Sun, Feb 14, 2016 at 7:04 PM, Duy Nguyen \n> wrote:\n> > On Sat, Feb 6, 2016 at 2:44 AM, David Turner <\n> > dturner@twopensource.com> wrote:\n> > > +static char *get_refdb_path(const char *base)\n> > > +{\n> > > + static struct strbuf path_buf = STRBUF_INIT;\n> > > + strbuf_reset(&path_buf);\n> > > + strbuf_addf(&path_buf, \"%s/refdb\", base);\n> > > + return path_buf.buf;\n> > > +}\n> > ...\n> > > +static int lmdb_init_db(struct strbuf *err, int shared)\n> > > +{\n> > > + /*\n> > > + * To create a db, all we need to do is make a directory\n> > > for\n> > > + * it to live in; lmdb will do the rest.\n> > > + */\n> > > +\n> > > + if (!db_path)\n> > > + db_path =\n> > > xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n> > \n> > This works for multiple worktrees. But scripts may have harder time\n> > getting the path. The recommended way is \"git rev-parse --git-path\n> > refdb\" but because \"refdb\" is not registered in\n> > path.c:common_list[],\n> > that command becomes git_path(\"refdb\") instead of\n> > get_refdb(get_git_... like here. And I will need to know that\n> > .git/refdb is _not_ per-worktree when I migrate/convert main\n> > worktree\n> > (it's very likely I have to go that route to solve .git/config\n> > issue\n> > in multi worktree).\n> > \n> > The solution is register refdb to common_list[] and you can do\n> > git_path(\"refdb\") here. But then what happens when another backend\n> > is\n> > added? Will the new backend use the same path \"refdb\", or say\n> > \"refdb.sqlite\"? If all backends share the name \"refdb\", why can't\n> > we\n> > just reuse \"refs\" instead because the default filesystem-based\n> > backend\n> > is technically just another backend?\n> \n> To answer myself: I forgot that there were per-worktree refs (e.g.\n> refs/bisect). It makes me wonder if we should put per-worktree refs\n> to\n> lmdb as well (maybe one per worktree if we don't want to put all in\n> one db). One of the advantages of moving away from fs-based backend\n> is\n> the ability to deal with case sensitivity, \"nested\" refs (e.g. a/b\n> and\n> a/b/c are both refs).\n\nI don't think case-sensitivity is a huge problem here: per-worktree\nrefs are now HEAD or refs/bisect/*, and refs/bisect is controlled\nentirely by the bisection machinery (that is, it won't have user\n-defined refs with case conflicts).\n\n> With this split, I think some refs are still\n> left behind.. Sorry if this was discussed before, I haven't followed\n> this closely.\n\nPrevious discussion:\nhttp://comments.gmane.org/gmane.comp.version-control.git/275097\n"},{"id":"278499","messageId":"1455741139.7528.14.camel@twopensource.com","threadId":"41344","inReplyTo":"CACsJy8CWS9E0Jb4omm4k0=kZ73w9UoHmofLFCFHUhHu53zfgYw@mail.gmail.com","subject":"Re: [PATCH v4 20/21] refs: add LMDB refs storage backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-17T20:32:19Z","receivedAt":"2016-02-17T20:32:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sun, 2016-02-14 at 19:04 +0700, Duy Nguyen wrote:\n> On Sat, Feb 6, 2016 at 2:44 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > +static char *get_refdb_path(const char *base)\n> > +{\n> > + static struct strbuf path_buf = STRBUF_INIT;\n> > + strbuf_reset(&path_buf);\n> > + strbuf_addf(&path_buf, \"%s/refdb\", base);\n> > + return path_buf.buf;\n> > +}\n> ...\n> > +static int lmdb_init_db(struct strbuf *err, int shared)\n> > +{\n> > + /*\n> > + * To create a db, all we need to do is make a directory\n> > for\n> > + * it to live in; lmdb will do the rest.\n> > + */\n> > +\n> > + if (!db_path)\n> > + db_path =\n> > xstrdup(real_path(get_refdb_path(get_git_common_dir())));\n> \n> This works for multiple worktrees. But scripts may have harder time\n> getting the path. The recommended way is \"git rev-parse --git-path\n> refdb\" but because \"refdb\" is not registered in path.c:common_list[],\n> that command becomes git_path(\"refdb\") instead of\n> get_refdb(get_git_... like here. And I will need to know that\n> .git/refdb is _not_ per-worktree when I migrate/convert main worktree\n> (it's very likely I have to go that route to solve .git/config issue\n> in multi worktree).\n\nI'll fix common_list. \n\n> The solution is register refdb to common_list[] and you can do\n> git_path(\"refdb\") here. But then what happens when another backend is\n> added? Will the new backend use the same path \"refdb\", or say\n> \"refdb.sqlite\"? If all backends share the name \"refdb\", why can't we\n> just reuse \"refs\" instead because the default filesystem-based \n> backend is technically just another backend?\n\nI'll change it to \"refs.lmdb\".\n\nI don't want to use \"refs\" because it makes it hard for software to\ndistinguish the format (in some cases, like sqlite, it won't even be a\ndirectory).\n"},{"id":"278500","messageId":"1455741172.7528.15.camel@twopensource.com","threadId":"41344","inReplyTo":"56BDFD4C.3050809@alum.mit.edu","subject":"Re: [PATCH v4 19/21] refs: add register_ref_storage_backends()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-17T20:32:52Z","receivedAt":"2016-02-17T20:32:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-02-12 at 16:42 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > This new function will register all known ref storage backends...\n> > once\n> > there are any other than the default. For now, it's a no-op.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > builtin/init-db.c | 3 +++\n> > config.c | 25 +++++++++++++++++++++++++\n> > refs.c | 8 ++++++++\n> > refs.h | 2 ++\n> > 4 files changed, 38 insertions(+)\n> > \n> > [...]\n> > diff --git a/config.c b/config.c\n> > index b95ac3a..b9ef223 100644\n> > --- a/config.c\n> > +++ b/config.c\n> > @@ -11,6 +11,7 @@\n> > #include \"strbuf.h\"\n> > #include \"quote.h\"\n> > #include \"hashmap.h\"\n> > +#include \"refs.h\"\n> > #include \"string-list.h\"\n> > #include \"utf8.h\"\n> > \n> > @@ -1207,6 +1208,30 @@ int git_config_early(config_fn_t fn, void\n> > *data, const char *repo_config)\n> > \t}\n> > \n> > \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n> > +\t\tchar *storage = NULL;\n> > +\n> > +\t\t/*\n> > +\t\t * make sure we always read the ref storage config\n> > +\t\t * from the extensions section on startup\n> > +\t\t */\n> > +\t\tret +=\n> > git_config_from_file(ref_storage_backend_config,\n> > +\t\t\t\t\t repo_config,\n> > &storage);\n> > +\n> > +\t\tregister_ref_storage_backends();\n> > +\t\tif (!storage)\n> > +\t\t\tstorage = xstrdup(\"\");\n> > +\n> > +\t\tif ((!*storage) ||\n> > +\t\t (!strcmp(storage, \"files\"))) {\n> \n> Nit: you have some unnecessary parentheses here.\n\nFixed, thanks.\n\n> > +\t\t\t/* default backend, nothing to do */\n> > +\t\t\tfree(storage);\n> > +\t\t} else {\n> > +\t\t\tref_storage_backend = storage;\n> > +\t\t\tif\n> > (set_ref_storage_backend(ref_storage_backend))\n> > +\t\t\t\tdie(_(\"Unknown ref storage backend\n> > %s\"),\n> > +\t\t\t\t ref_storage_backend);\n> > +\t\t}\n> > +\n> > \t\tret += git_config_from_file(fn, repo_config,\n> > data);\n> > \t\tfound += 1;\n> > \t}\n> > diff --git a/refs.c b/refs.c\n> > index 715a492..e50cca0 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -1554,3 +1554,11 @@ done:\n> > \tstring_list_clear(&affected_refnames, 0);\n> > \treturn ret;\n> > }\n> > +\n> > +void register_ref_storage_backends(void) {\n> > +\t/*\n> > +\t * No need to register the files backend; it's registered\n> > by\n> > +\t * default. Add register_ref_storage_backend(ptr-to\n> > -backend)\n> > +\t * entries below when you add a new backend.\n> > +\t */\n> \n> This function must be called every run, right? So why not make it\n> register the \"files\" backend explicitly? That would make it obvious\n> really quick if this function fails to get called in some code path.\n> Just a thought...\n\nGood idea.\n"},{"id":"278503","messageId":"1455742036.7528.18.camel@twopensource.com","threadId":"41344","inReplyTo":"56BDF9B1.5050302@alum.mit.edu","subject":"Re: [PATCH v4 15/21] init: allow alternate ref strorage to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-17T20:47:16Z","receivedAt":"2016-02-17T20:47:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-02-12 at 16:26 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > git init learns a new argument --ref-storage. Presently, only\n> > \"files\" is supported, but later we will add other storage backends.\n> > \n> > When this argument is used, the repository's extensions.refStorage\n> > configuration value is set (as well as\n> > core.repositoryformatversion),\n> > and the ref storage backend's initdb function is used to set up the\n> > ref database.\n> > \n> > Signed-off-by: David Turner \n> > Signed-off-by: SZEDER Gábor \n> > ---\n> > Documentation/git-init-db.txt | 2 +-\n> > Documentation/git-init.txt | 7 +++++-\n> > builtin/init-db.c | 44\n> > +++++++++++++++++++++++++++-------\n> > cache.h | 2 ++\n> > contrib/completion/git-completion.bash | 3 ++-\n> > path.c | 29 ++++++++++++++++++++--\n> > refs.c | 8 +++++++\n> > refs.h | 8 +++++++\n> > setup.c | 12 ++++++++++\n> > t/t0001-init.sh | 25 +++++++++++++++++++\n> > 10 files changed, 127 insertions(+), 13 deletions(-)\n> > \n> > [...]\n> > diff --git a/Documentation/git-init.txt b/Documentation/git\n> > -init.txt\n> > index 8174d27..d2b150f 100644\n> > --- a/Documentation/git-init.txt\n> > +++ b/Documentation/git-init.txt\n> > @@ -12,7 +12,7 @@ SYNOPSIS\n> > 'git init' [-q | --quiet] [--bare] [-\n> > -template=]\n> > \t [--separate-git-dir ]\n> > \t [--shared[=]] [directory]\n> > -\n> > +\t [--ref-storage=]\n> > \n> > DESCRIPTION\n> > -----------\n> > @@ -113,6 +113,11 @@ does not exist, it will be created.\n> > \n> > --\n> > \n> > +--ref-storage=::\n> > +Type of refs storage backend. Default is to use the original\n> > \"files\"\n> > +storage, which stores ref data in files in .git/refs and\n> > +.git/packed-refs.\n> > +\n> \n> Technically, that should be $GIT_DIR/refs and $GIT_DIR/packed-refs.\n> But\n> it might be that we are not so picky about the distinction in the\n> user docs.\n\nWe're not (according to grep), but we probably should be.\n\n> > TEMPLATE DIRECTORY\n> > ------------------\n> > \n> > diff --git a/builtin/init-db.c b/builtin/init-db.c\n> > index 801e977..d331ce8 100644\n> > --- a/builtin/init-db.c\n> > +++ b/builtin/init-db.c\n> > @@ -24,6 +24,7 @@ static int init_is_bare_repository = 0;\n> > static int init_shared_repository = -1;\n> > static const char *init_db_template_dir;\n> > static const char *git_link;\n> > +static char *requested_ref_storage_backend;\n> > \n> > static void copy_templates_1(struct strbuf *path, struct strbuf\n> > *template,\n> > \t\t\t DIR *dir)\n> > @@ -179,6 +180,7 @@ static int create_default_files(const char\n> > *template_path)\n> > \tint reinit;\n> > \tint filemode;\n> > \tstruct strbuf err = STRBUF_INIT;\n> > +\tint repo_version = 0;\n> > \n> > \t/* Just look for `init.templatedir` */\n> > \tgit_config(git_init_db_config, NULL);\n> > @@ -205,6 +207,32 @@ static int create_default_files(const char\n> > *template_path)\n> > \t}\n> > \n> > \t/*\n> > +\t * Create the default symlink from \".git/HEAD\" to the\n> > \"master\"\n> \n> I know that you are just moving this comment, but s/symlink/symref/\n> would make it more up-to-date.\n\nOK.\n\n> > +\t * branch, if it does not exist yet.\n> > +\t */\n> > [...]\n> > diff --git a/setup.c b/setup.c\n> > index 0deb022..1a62277 100644\n> > --- a/setup.c\n> > +++ b/setup.c\n> > @@ -1,5 +1,6 @@\n> > #include \"cache.h\"\n> > #include \"dir.h\"\n> > +#include \"refs.h\"\n> > #include \"string-list.h\"\n> > \n> > static int inside_git_dir = -1;\n> > @@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb,\n> > const char *gitdir)\n> > \treturn ret;\n> > }\n> > \n> > +int ref_storage_backend_config(const char *var, const char *value,\n> > void *ptr)\n> > +{\n> > +\tchar **cdata = ptr;\n> > +\n> > +\tif (!strcmp(var, \"extensions.refstorage\"))\n> > +\t\t*cdata = xstrdup(value);\n> > +\treturn 0;\n> > +}\n> > +\n> > /*\n> > * Test if it looks like we're at a git directory.\n> > * We want to see:\n> > @@ -390,6 +400,8 @@ static int check_repo_format(const char *var,\n> > const char *value, void *cb)\n> > \t\t\t;\n> > \t\telse if (!strcmp(ext, \"preciousobjects\"))\n> > \t\t\trepository_format_precious_objects =\n> > git_config_bool(var, value);\n> > +\t\telse if (!strcmp(ext, \"refstorage\"))\n> > +\t\t\tref_storage_backend = xstrdup(value);\n> > \t\telse\n> > \t\t\tstring_list_append(&unknown_extensions,\n> > ext);\n> > \t}\n> > diff --git a/t/t0001-init.sh b/t/t0001-init.sh\n> > index 295aa59..31c8279 100755\n> > --- a/t/t0001-init.sh\n> > +++ b/t/t0001-init.sh\n> > @@ -174,6 +174,31 @@ test_expect_success 'reinit' '\n> > \ttest_i18ncmp again/empty again/err2\n> > '\n> > \n> > +test_expect_success 'init with bogus storage backend fails' '\n> > +\n> > +\t(\n> > +\t\tmkdir again2 &&\n> > +\t\tcd again2 &&\n> > +\t\ttest_must_fail git init --ref-storage=test >out2\n> > 2>err2 &&\n> > +\t\ttest_i18ngrep \"Unknown ref storage backend test\"\n> > err2\n> > +\t)\n> > +'\n> > +\n> > +test_expect_failure 'reinit with changed storage backend fails' '\n> > +\n> > +\t(\n> > +\t\tmkdir again3 &&\n> > +\t\tcd again3 &&\n> > +\t\tgit init >out1 2>err1 &&\n> > +\t\tgit init --ref-storage=test >out2 2>err2\n> > +\t) &&\n> > +\ttest_i18ngrep \"Initialized empty\" again3/out1 &&\n> > +\ttest_i18ngrep \"Reinitialized existing\" again3/out2 &&\n> > +\t>again3/empty &&\n> > +\ttest_i18ncmp again3/empty again3/err1 &&\n> > +\ttest_i18ncmp again3/empty again3/err2\n> > +'\n> > +\n> \n> Is it worth testing re-initializing with the same --ref-storage?\n> Perhaps\n> not.\n\nWould be nice, but since we cannot guarantee that any alternate backend\nexists, we have no way to test this. We'd have to add an entire \"test\"\nref backend just for tests, which seems a bit overboard.\n\n> > test_expect_success 'init with --template' '\n> > \tmkdir template-source &&\n> > \techo content >template-source/file &&\n> > \n> \n> Michael\n> \n"},{"id":"278523","messageId":"1455755367.7528.57.camel@twopensource.com","threadId":"41344","inReplyTo":"56BDE780.2030407@alum.mit.edu","subject":"Re: [PATCH v4 13/21] refs: resolve symbolic refs first","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-18T00:29:27Z","receivedAt":"2016-02-18T00:29:27Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 201-02-12 at 15:09 +0100, Michael Haggerty wrote:]\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Before committing ref updates, split symbolic ref updates into two\n> > parts: an update to the underlying ref, and a log-only update to\n> > the\n> > symbolic ref. This ensures that both references are locked\n> > correctly\n> > while their reflogs are updated.\n> > \n> > It is still possible to confuse git by concurrent updates, since\n> > the\n> > splitting of symbolic refs does not happen under lock. So a\n> > symbolic ref\n> > could be replaced by a plain ref in the middle of this operation,\n> > which\n> > would lead to reflog discontinuities and missed old-ref checks.\n> \n> This patch is doing too much at once for my little brain to follow.\n> \n> My first hangup is the change to setting RESOLVE_REF_NO_RECURSE\n> unconditionally in lock_ref_sha1_basic(). I count five callers of\n> that\n> function and see no justification for why the change is OK in the\n> context of each caller. Here are some thoughts:\n> \n> * The call from files_create_symref() sets REF_NODEREF, so it is\n> unaffected by this change.\n\nYes.\n\n> * The call from files_transaction_commit() is preceded by a call to\n> dereference_symrefs(), which I assume effectively replaces the need\n> for\n> RESOLVE_REF_NO_RECURSE.\n\nYes.\n\n> * There are two calls from files_rename_ref(). Why is it OK to do\n> without RESOLVE_REF_NO_RECURSE there?\n> \n> * For the oldrefname call, I suppose the justification is the\n> \"(flag &\n> REF_ISSYMREF)\" check earlier in the function. (But does this\n> introduce a\n> significant TOCTOU race?)\n\nThe refs code as a whole seems likely to have TOCTOU issues. In\ngeneral, anywhere we check/set flag & REF_ISSYMREF without holding a\nlock, we have a potential problem. I haven't generally tried to handle\nthese cases, since they're not presently handled. \n\nThe central problem with this area of the code is that commit interacts\nso intimately with the locking machinery. I understand some of why\nit's done that way. In particular, your change to ref locking to not\nhold lots of open files was a big win for us at Twitter. But this\nmeans that it's hard to deal with cross-backend ref updates: you want\nto hold multiple locks, and backends don't have the machinery for it.\n\nWe could add backend hooks to specifically lock and unlock refs. Then\nthe backend commit code would just be handled a bundle of locked refs\nand would commit them. This might be hairy, but it could fix the\nTOCTOU problems. So, first lock the outer refs, then split out updates\nfor any which are symbolic refs, and lock those. Finally, commit all\nupdates (split by backend).\n\nOne downside of this is that right now, the backend API is relatively\nclose to the front-end, and this would leak what should be an\nimplementation detail. But maybe this is necessary to knit multiple\nbackends together. \n\nBut I'm not sure that this is necessary right now, because I'm not sure\nthat I'm actually making TOCTOU issues much worse. \n\n> * For the newrefname call, I suppose it's because the code a little\n> higher up tries to delete any existing reference with that name. It\n> looks to me like the old code was slightly broken: if newrefname was\n> an\n> unborn symbolic reference, then: read_ref_full() would fail;\n> delete_ref() would be skipped; lock_ref_sha1_basic() would lock the\n> *referred-to* reference; the referred-to reference would be\n> overwritten\n> instead of newrefname. So it could be that here REF_NODEREF\n> indirectly\n> fixes a bug?\n\nYes, that's correct. These two appears to be separable, so I'll make\nit an independent patch (and add a test for that case).\n\n> * The last call, from files_reflog_expire(), is also questionable \n> before your patch. If refname is a symref, then the function is \n> expiring the reflog of the symref. But (before this patch) it locks \n> not the symref but its referent. \n\nI can also separate this one.\n\n> This was discussed in some length before on the mailing list [1], and\n> the conclusion was that the current behavior is wrong, but for\n> backwards compatibility reasons it would be safest to change it to\n> locking *both* the symref and its referent.\n\nYes, that would be the right thing to do. But for the reasons I\ndiscuss above, that requires a serious change in the way that backends\nwork. \n\n> If possible, it would be better to split this patch up into several:\n> the\n> first few would each add the REF_NODEREF flag at one callsite, with a\n> careful justification of why that is OK. Once all the callsites\n> (except\n> the one in files_transaction_commit()) have been changed, then the\n> last\n> patch could add the dereference_symrefs() machinery and change the\n> last\n> callsite.\n> \n> (I'm not certain that those steps are actually doable independently,\n> given that REF_NODEREF has other effects besides setting\n> RESOLVE_REF_NO_RECURSE.)\n> \n> I'm not just being pedantic here. The patch as written is really too\n> big\n> to review effectively.\n\nThat's a legit complaint. The problem, as you note, is that doing some\nof these steps completely independently doesn't work. But I'll try\nsplitting out what I can.\n"},{"id":"278525","messageId":"1455763440.7528.71.camel@twopensource.com","threadId":"41344","inReplyTo":"56BDF531.7010909@alum.mit.edu","subject":"Re: [PATCH v4 14/21] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-18T02:44:00Z","receivedAt":"2016-02-18T02:44:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-02-12 at 16:07 +0100, Michael Haggerty wrote:\n> On 02/05/2016 08:44 PM, David Turner wrote:\n> > Always handle non-normal (per-worktree or pseudo) refs in the files\n> > backend instead of alternate backends.\n> > \n> > Sometimes a ref transaction will update both a per-worktree ref and\n> > a\n> > normal ref. For instance, an ordinary commit might update\n> > refs/heads/master and HEAD (or at least HEAD's reflog).\n> > \n> > Updates to normal refs continue to go through the chosen backend.\n> > \n> > Updates to non-normal refs are moved to a separate files backend\n> > transaction.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > refs.c | 81\n> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n> > 1 file changed, 79 insertions(+), 2 deletions(-)\n> > \n> > diff --git a/refs.c b/refs.c\n> > index 227c018..18ba356 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -9,6 +9,11 @@\n> > #include \"object.h\"\n> > #include \"tag.h\"\n> > \n> > +static const char split_transaction_fail_warning[] = N_(\n> > +\t\"A ref transaction was split across two refs backends. \n> > Part of the \"\n> > +\t\"transaction succeeded, but then the update to the per\n> > -worktree refs \"\n> > +\t\"failed. Your repository may be in an inconsistent\n> > state.\");\n> > +\n> > /*\n> > * We always have a files backend and it is the default.\n> > */\n> > @@ -791,6 +796,13 @@ void ref_transaction_free(struct\n> > ref_transaction *transaction)\n> > \tfree(transaction);\n> > }\n> > \n> > +static void add_update_obj(struct ref_transaction *transaction,\n> > +\t\t\t struct ref_update *update)\n> > +{\n> > +\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n> > transaction->alloc);\n> > +\ttransaction->updates[transaction->nr++] = update;\n> > +}\n> > +\n> > static struct ref_update *add_update(struct ref_transaction\n> > *transaction,\n> > \t\t\t\t const char *refname)\n> > {\n> > @@ -798,8 +810,7 @@ static struct ref_update *add_update(struct\n> > ref_transaction *transaction,\n> > \tstruct ref_update *update = xcalloc(1, sizeof(*update) +\n> > len);\n> > \n> > \tmemcpy((char *)update->refname, refname, len); /* includes\n> > NUL */\n> > -\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n> > transaction->alloc);\n> > -\ttransaction->updates[transaction->nr++] = update;\n> > +\tadd_update_obj(transaction, update);\n> > \treturn update;\n> > }\n> > \n> > @@ -1217,11 +1228,38 @@ static int dereference_symrefs(struct\n> > ref_transaction *transaction,\n> > \treturn 0;\n> > }\n> > \n> > +/*\n> > + * Move all non-normal ref updates into a specially-created\n> > + * files-backend transaction\n> > + */\n> > +static int move_abnormal_ref_updates(struct ref_transaction\n> > *transaction,\n> > +\t\t\t\t struct ref_transaction\n> > *files_transaction,\n> > +\t\t\t\t struct strbuf *err)\n> > +{\n> > +\tint i;\n> > +\n> > +\tfor (i = 0; i < transaction->nr; i++) {\n> > +\t\tint last;\n> > +\t\tstruct ref_update *update = transaction\n> > ->updates[i];\n> > +\n> > +\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n> > +\t\t\tcontinue;\n> > +\n> > +\t\tlast = --transaction->nr;\n> > +\t\ttransaction->updates[i] = transaction\n> > ->updates[last];\n> > +\t\tadd_update_obj(files_transaction, update);\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> \n> I think this function is incorrect. The update that was previously at\n> transaction->updates[i] never gets checked for abnormality. \n\nYes it does; that's the \"update\" variable that we just checked.\n\n> You could\n> fix that and also avoid gratuitously changing the order of the\n> updates\n> like this:\n> \n> static int move_abnormal_ref_updates(struct ref_transaction\n> *transaction,\n> \t\t\t\t struct ref_transaction\n> *files_transaction,\n> \t\t\t\t struct strbuf *err)\n> {\n> \tint i, normal_nr = 0;\n> \n> \tfor (i = 0; i < transaction->nr; i++) {\n> \t\tstruct ref_update *update = transaction->updates[i];\n> \n> \t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n> \t\t\ttransaction->updates[normal_nr++] = update;\n> \t\telse\n> \t\t\tadd_update_obj(files_transaction, update);\n> \t}\n> \n> \ttransaction->nr = normal_nr;\n> \treturn 0;\n> }\n\nSure, I can make that change.\n\n> Another alternative would be to set\n> \n> update->flags |= REF_ABNORMAL\n> \n> for the abnormal references and *leave them* in the original\n> transaction\n> while also adding them to files_transactions. Then teach non-files\n> backends to skip over updates with REF_ABNORMAL.\n> \n> The reason I thought of this is that long-term I'd like to make it\n> possible for some reference updates to fail without aborting the\n> transaction. To implement that, we would want a way for\n> ref_transaction_commit() to report back to its caller *which* updates\n> failed, and an obvious way to do that would be to store the\n> reference-specific errors in struct ref_update. If you leave the\n> abnormal ref_updates in the main transaction, then my hoped-for\n> change\n> would be easier.\n> \n> But that's a separate and hypothetical idea, so you don't have to let\n> it\n> influence how you implement this patch.\n\nI'm also interested in this idea. Perhaps it would also be nice to\nreport *why* they fail (e.g. the conflicting ref name). I did a\nvariant of this with for the journal code, but my way of doing it\nturned out to be a bad idea (long story). But I want to stay focused\non the simplest thing possible, for now.\n\n> > int ref_transaction_commit(struct ref_transaction *transaction,\n> > \t\t\t struct strbuf *err)\n> > {\n> > \tint ret = -1;\n> > \tstruct string_list affected_refnames =\n> > STRING_LIST_INIT_NODUP;\n> > +\tstruct string_list files_affected_refnames =\n> > STRING_LIST_INIT_NODUP;\n> > +\tstruct ref_transaction *files_transaction = NULL;\n> > \n> > \tassert(err);\n> > \n> > @@ -1237,6 +1275,26 @@ int ref_transaction_commit(struct\n> > ref_transaction *transaction,\n> > \tif (ret)\n> > \t\tgoto done;\n> > \n> > +\tif (the_refs_backend != &refs_be_files) {\n> > +\t\tfiles_transaction = ref_transaction_begin(err);\n> > +\t\tif (!files_transaction)\n> > +\t\t\tgoto done;\n> > +\n> > +\t\tret = move_abnormal_ref_updates(transaction,\n> > files_transaction,\n> > +\t\t\t\t\t\terr);\n> > +\t\tif (ret)\n> > +\t\t\tgoto done;\n> > +\n> > +\t\t/* files backend commit */\n> > +\t\tif (get_affected_refnames(files_transaction,\n> > +\t\t\t\t\t\t &files_affected_r\n> > efnames,\n> > +\t\t\t\t\t\t err)) {\n> > +\t\t\tret = TRANSACTION_GENERIC_ERROR;\n> > +\t\t\tgoto done;\n> > +\t\t}\n> > +\t}\n> > +\n> > +\t/* main backend commit */\n> > \tif (get_affected_refnames(transaction, &affected_refnames,\n> > err)) {\n> > \t\tret = TRANSACTION_GENERIC_ERROR;\n> > \t\tgoto done;\n> > @@ -1244,8 +1302,24 @@ int ref_transaction_commit(struct\n> > ref_transaction *transaction,\n> > \n> > \tret = the_refs_backend->transaction_commit(transaction,\n> > \t\t\t\t\t\t \n> > &affected_refnames, err);\n> > +\tif (ret)\n> > +\t\tgoto done;\n> > +\n> > +\tif (files_transaction) {\n> > +\t\tret =\n> > refs_be_files.transaction_commit(files_transaction,\n> > +\t\t\t\t\t\t \n> > &files_affected_refnames,\n> > +\t\t\t\t\t\t err);\n> > +\t\tif (ret) {\n> > +\t\t\twarning(split_transaction_fail_warning);\n> > +\t\t\tgoto done;\n> > +\t\t}\n> > +\t}\n> > +\n> > done:\n> > \tstring_list_clear(&affected_refnames, 0);\n> > +\tstring_list_clear(&files_affected_refnames, 0);\n> > +\tif (files_transaction)\n> > +\t\tref_transaction_free(files_transaction);\n> > \treturn ret;\n> > }\n> > \n> > @@ -1285,6 +1359,9 @@ int peel_ref(const char *refname, unsigned\n> > char *sha1)\n> > int create_symref(const char *ref_target, const char\n> > *refs_heads_master,\n> > \t\t const char *logmsg)\n> > {\n> > +\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n> > +\t\treturn refs_be_files.create_symref(ref_target,\n> > refs_heads_master,\n> > +\t\t\t\t\t\t logmsg);\n> > \treturn the_refs_backend->create_symref(ref_target,\n> > refs_heads_master,\n> > \t\t\t\t\t logmsg);\n> > }\n> > \n> \n> Does initial_ref_transaction_commit() need the same treatment?\n\nWe only use that for remote refs -- I'm not sure if those can be\nsymrefs. Wouldn't hurt.\n\n> I think files_rename_ref() will break if one of the references is\n> normal\n> and one is abnormal. I think it would be OK to prohibit renaming\n> abnormal refs entirely (can anybody think of an important use case?),\n> but that function should at least do the checks.\n\nThe files backend would work fine, but I guess I'll add the check\nanyway, since that would be a weird thing to do. \n"},{"id":"278559","messageId":"56C5B23E.6090905@alum.mit.edu","threadId":"41344","inReplyTo":"1455755367.7528.57.camel@twopensource.com","subject":"Re: [PATCH v4 13/21] refs: resolve symbolic refs first","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-18T11:59:58Z","receivedAt":"2016-02-18T11:59:58Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/18/2016 01:29 AM, David Turner wrote:\n> On Fri, 201-02-12 at 15:09 +0100, Michael Haggerty wrote:]\n>> On 02/05/2016 08:44 PM, David Turner wrote:\n>>> Before committing ref updates, split symbolic ref updates into two\n>>> parts: an update to the underlying ref, and a log-only update to\n>>> the\n>>> symbolic ref. This ensures that both references are locked\n>>> correctly\n>>> while their reflogs are updated.\n>>>\n>>> It is still possible to confuse git by concurrent updates, since\n>>> the\n>>> splitting of symbolic refs does not happen under lock. So a\n>>> symbolic ref\n>>> could be replaced by a plain ref in the middle of this operation,\n>>> which\n>>> would lead to reflog discontinuities and missed old-ref checks.\n>>\n>> This patch is doing too much at once for my little brain to follow.\n>>\n>> My first hangup is the change to setting RESOLVE_REF_NO_RECURSE\n>> unconditionally in lock_ref_sha1_basic(). I count five callers of\n>> that\n>> function and see no justification for why the change is OK in the\n>> context of each caller. Here are some thoughts:\n>>\n>> * The call from files_create_symref() sets REF_NODEREF, so it is\n>> unaffected by this change.\n> \n> Yes.\n> \n>> * The call from files_transaction_commit() is preceded by a call to\n>> dereference_symrefs(), which I assume effectively replaces the need\n>> for\n>> RESOLVE_REF_NO_RECURSE.\n> \n> Yes.\n> \n>> * There are two calls from files_rename_ref(). Why is it OK to do\n>> without RESOLVE_REF_NO_RECURSE there?\n>>\n>> * For the oldrefname call, I suppose the justification is the\n>> \"(flag &\n>> REF_ISSYMREF)\" check earlier in the function. (But does this\n>> introduce a\n>> significant TOCTOU race?)\n> \n> The refs code as a whole seems likely to have TOCTOU issues. In\n> general, anywhere we check/set flag & REF_ISSYMREF without holding a\n> lock, we have a potential problem. I haven't generally tried to handle\n> these cases, since they're not presently handled. \n\nI agree that we don't do so well here, though I think that most races\nwould result in reading/writing a ref that was pointed to by the symref\na moment ago, which is usually indistinguishable to the user from their\nupdate having gone through the moment before the symref was updated. So\nI don't think your change makes this bit of code significantly worse.\n\n> The central problem with this area of the code is that commit interacts\n> so intimately with the locking machinery. I understand some of why\n> it's done that way. In particular, your change to ref locking to not\n> hold lots of open files was a big win for us at Twitter. But this\n> means that it's hard to deal with cross-backend ref updates: you want\n> to hold multiple locks, and backends don't have the machinery for it.\n> \n> We could add backend hooks to specifically lock and unlock refs. Then\n> the backend commit code would just be handled a bundle of locked refs\n> and would commit them. This might be hairy, but it could fix the\n> TOCTOU problems. So, first lock the outer refs, then split out updates\n> for any which are symbolic refs, and lock those. Finally, commit all\n> updates (split by backend).\n\nAs chance would have it, for an internal GitHub project I've implemented\nhooks that can be called *during* a ref transaction. The hooks can, for\nexample, take arbitrary actions between the time that the reflocks are\nall acquired and the time that the updates start to be committed. I\ndidn't submit this code upstream because I didn't think that it would\nbenefit other users, but many it would be useful for implementing\nsplit-backend reference transaction commits. E.g., the primary reference\ntransaction could run the secondary backend's commit while holding the\nlocks for the primary backend references.\n\nLet me think about it.\n\nI don't think this is urgent though. The current code is not\nsignificantly racy in mainstream usage scenarios, right?\n\n> One downside of this is that right now, the backend API is relatively\n> close to the front-end, and this would leak what should be an\n> implementation detail. But maybe this is necessary to knit multiple\n> backends together. \n> \n> But I'm not sure that this is necessary right now, because I'm not sure\n> that I'm actually making TOCTOU issues much worse. \n\nAgreed.\n\n> [...]\n> That's a legit complaint. The problem, as you note, is that doing some\n> of these steps completely independently doesn't work. But I'll try\n> splitting out what I can.\n\nThanks!\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278560","messageId":"56C5B41E.50502@alum.mit.edu","threadId":"41344","inReplyTo":"1455763440.7528.71.camel@twopensource.com","subject":"Re: [PATCH v4 14/21] refs: always handle non-normal refs in files backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-18T12:07:58Z","receivedAt":"2016-02-18T12:07:58Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/18/2016 03:44 AM, David Turner wrote:\n> On Fri, 2016-02-12 at 16:07 +0100, Michael Haggerty wrote:\n>> On 02/05/2016 08:44 PM, David Turner wrote:\n>>> Always handle non-normal (per-worktree or pseudo) refs in the files\n>>> backend instead of alternate backends.\n>>>\n>>> Sometimes a ref transaction will update both a per-worktree ref and\n>>> a\n>>> normal ref. For instance, an ordinary commit might update\n>>> refs/heads/master and HEAD (or at least HEAD's reflog).\n>>>\n>>> Updates to normal refs continue to go through the chosen backend.\n>>>\n>>> Updates to non-normal refs are moved to a separate files backend\n>>> transaction.\n>>>\n>>> Signed-off-by: David Turner \n>>> ---\n>>> refs.c | 81\n>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n>>> 1 file changed, 79 insertions(+), 2 deletions(-)\n>>>\n>>> diff --git a/refs.c b/refs.c\n>>> index 227c018..18ba356 100644\n>>> --- a/refs.c\n>>> +++ b/refs.c\n>>> @@ -9,6 +9,11 @@\n>>> #include \"object.h\"\n>>> #include \"tag.h\"\n>>> \n>>> +static const char split_transaction_fail_warning[] = N_(\n>>> +\t\"A ref transaction was split across two refs backends. \n>>> Part of the \"\n>>> +\t\"transaction succeeded, but then the update to the per\n>>> -worktree refs \"\n>>> +\t\"failed. Your repository may be in an inconsistent\n>>> state.\");\n>>> +\n>>> /*\n>>> * We always have a files backend and it is the default.\n>>> */\n>>> @@ -791,6 +796,13 @@ void ref_transaction_free(struct\n>>> ref_transaction *transaction)\n>>> \tfree(transaction);\n>>> }\n>>> \n>>> +static void add_update_obj(struct ref_transaction *transaction,\n>>> +\t\t\t struct ref_update *update)\n>>> +{\n>>> +\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n>>> transaction->alloc);\n>>> +\ttransaction->updates[transaction->nr++] = update;\n>>> +}\n>>> +\n>>> static struct ref_update *add_update(struct ref_transaction\n>>> *transaction,\n>>> \t\t\t\t const char *refname)\n>>> {\n>>> @@ -798,8 +810,7 @@ static struct ref_update *add_update(struct\n>>> ref_transaction *transaction,\n>>> \tstruct ref_update *update = xcalloc(1, sizeof(*update) +\n>>> len);\n>>> \n>>> \tmemcpy((char *)update->refname, refname, len); /* includes\n>>> NUL */\n>>> -\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n>>> transaction->alloc);\n>>> -\ttransaction->updates[transaction->nr++] = update;\n>>> +\tadd_update_obj(transaction, update);\n>>> \treturn update;\n>>> }\n>>> \n>>> @@ -1217,11 +1228,38 @@ static int dereference_symrefs(struct\n>>> ref_transaction *transaction,\n>>> \treturn 0;\n>>> }\n>>> \n>>> +/*\n>>> + * Move all non-normal ref updates into a specially-created\n>>> + * files-backend transaction\n>>> + */\n>>> +static int move_abnormal_ref_updates(struct ref_transaction\n>>> *transaction,\n>>> +\t\t\t\t struct ref_transaction\n>>> *files_transaction,\n>>> +\t\t\t\t struct strbuf *err)\n>>> +{\n>>> +\tint i;\n>>> +\n>>> +\tfor (i = 0; i < transaction->nr; i++) {\n>>> +\t\tint last;\n>>> +\t\tstruct ref_update *update = transaction\n>>> ->updates[i];\n>>> +\n>>> +\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n>>> +\t\t\tcontinue;\n>>> +\n>>> +\t\tlast = --transaction->nr;\n>>> +\t\ttransaction->updates[i] = transaction\n>>> ->updates[last];\n>>> +\t\tadd_update_obj(files_transaction, update);\n>>> +\t}\n>>> +\n>>> +\treturn 0;\n>>> +}\n>>> +\n>>\n>> I think this function is incorrect. The update that was previously at\n>> transaction->updates[i] never gets checked for abnormality. \n> \n> Yes it does; that's the \"update\" variable that we just checked.\n\nSorry, I meant to say \"the update that was previously at\n`transaction->updates[transaction->nr - 1]` never gets checked for\nabnormality\". Because it gets moved to `transaction->updates[i]`, then\n`i` is incremented without checking it.\n\n> [...]\n>> Another alternative would be to set\n>>\n>> update->flags |= REF_ABNORMAL\n>> [...]\n> I'm also interested in this idea. Perhaps it would also be nice to\n> report *why* they fail (e.g. the conflicting ref name). I did a\n> variant of this with for the journal code, but my way of doing it\n> turned out to be a bad idea (long story). But I want to stay focused\n> on the simplest thing possible, for now.\n\nFair enough.\n\n> [...]\n>> Does initial_ref_transaction_commit() need the same treatment?\n> \n> We only use that for remote refs -- I'm not sure if those can be\n> symrefs. Wouldn't hurt.\n\nHmmm, good point. I think the only symrefs that can be set during a\nclone are `HEAD` and `refs/remotes/origin/HEAD`. But I guess that no\nother references are updated *through* these symrefs, so it's probably\nOK. I haven't checked carefully, though.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278561","messageId":"56C5D156.1020800@alum.mit.edu","threadId":"41344","inReplyTo":"1455742036.7528.18.camel@twopensource.com","subject":"Re: [PATCH v4 15/21] init: allow alternate ref strorage to be set for new repos","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-02-18T14:12:38Z","receivedAt":"2016-02-18T14:12:38Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/17/2016 09:47 PM, David Turner wrote:\n> On Fri, 2016-02-12 at 16:26 +0100, Michael Haggerty wrote:\n>> [...]\n>> Is it worth testing re-initializing with the same --ref-storage?\n>> Perhaps\n>> not.\n> \n> Would be nice, but since we cannot guarantee that any alternate backend\n> exists, we have no way to test this. We'd have to add an entire \"test\"\n> ref backend just for tests, which seems a bit overboard.\n\nI meant re-initializing a repository that is using the \"files\" backend\nwith `--ref-storage=files` to make sure that it is a NOP. But I think\nthis functionality is unlikely to be broken so it wouldn't be a very\nhigh-value test.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"278565","messageId":"1455820357.7528.72.camel@twopensource.com","threadId":"41344","inReplyTo":"56C5B41E.50502@alum.mit.edu","subject":"Re: [PATCH v4 14/21] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-02-18T18:32:37Z","receivedAt":"2016-02-18T18:32:37Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-02-18 at 13:07 +0100, Michael Haggerty wrote:\n> On 02/18/2016 03:44 AM, David Turner wrote:\n> > On Fri, 2016-02-12 at 16:07 +0100, Michael Haggerty wrote:\n> > > On 02/05/2016 08:44 PM, David Turner wrote:\n> > > > Always handle non-normal (per-worktree or pseudo) refs in the\n> > > > files\n> > > > backend instead of alternate backends.\n> > > > \n> > > > Sometimes a ref transaction will update both a per-worktree ref\n> > > > and\n> > > > a\n> > > > normal ref. For instance, an ordinary commit might update\n> > > > refs/heads/master and HEAD (or at least HEAD's reflog).\n> > > > \n> > > > Updates to normal refs continue to go through the chosen\n> > > > backend.\n> > > > \n> > > > Updates to non-normal refs are moved to a separate files\n> > > > backend\n> > > > transaction.\n> > > > \n> > > > Signed-off-by: David Turner \n> > > > ---\n> > > > refs.c | 81\n> > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> > > > +--\n> > > > 1 file changed, 79 insertions(+), 2 deletions(-)\n> > > > \n> > > > diff --git a/refs.c b/refs.c\n> > > > index 227c018..18ba356 100644\n> > > > --- a/refs.c\n> > > > +++ b/refs.c\n> > > > @@ -9,6 +9,11 @@\n> > > > #include \"object.h\"\n> > > > #include \"tag.h\"\n> > > > \n> > > > +static const char split_transaction_fail_warning[] = N_(\n> > > > +\t\"A ref transaction was split across two refs backends.\n> > > > Part of the \"\n> > > > +\t\"transaction succeeded, but then the update to the per\n> > > > -worktree refs \"\n> > > > +\t\"failed. Your repository may be in an inconsistent\n> > > > state.\");\n> > > > +\n> > > > /*\n> > > > * We always have a files backend and it is the default.\n> > > > */\n> > > > @@ -791,6 +796,13 @@ void ref_transaction_free(struct\n> > > > ref_transaction *transaction)\n> > > > \tfree(transaction);\n> > > > }\n> > > > \n> > > > +static void add_update_obj(struct ref_transaction\n> > > > *transaction,\n> > > > +\t\t\t struct ref_update *update)\n> > > > +{\n> > > > +\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n> > > > transaction->alloc);\n> > > > +\ttransaction->updates[transaction->nr++] = update;\n> > > > +}\n> > > > +\n> > > > static struct ref_update *add_update(struct ref_transaction\n> > > > *transaction,\n> > > > \t\t\t\t const char *refname)\n> > > > {\n> > > > @@ -798,8 +810,7 @@ static struct ref_update *add_update(struct\n> > > > ref_transaction *transaction,\n> > > > \tstruct ref_update *update = xcalloc(1, sizeof(*update)\n> > > > +\n> > > > len);\n> > > > \n> > > > \tmemcpy((char *)update->refname, refname, len); /*\n> > > > includes\n> > > > NUL */\n> > > > -\tALLOC_GROW(transaction->updates, transaction->nr + 1,\n> > > > transaction->alloc);\n> > > > -\ttransaction->updates[transaction->nr++] = update;\n> > > > +\tadd_update_obj(transaction, update);\n> > > > \treturn update;\n> > > > }\n> > > > \n> > > > @@ -1217,11 +1228,38 @@ static int dereference_symrefs(struct\n> > > > ref_transaction *transaction,\n> > > > \treturn 0;\n> > > > }\n> > > > \n> > > > +/*\n> > > > + * Move all non-normal ref updates into a specially-created\n> > > > + * files-backend transaction\n> > > > + */\n> > > > +static int move_abnormal_ref_updates(struct ref_transaction\n> > > > *transaction,\n> > > > +\t\t\t\t struct ref_transaction\n> > > > *files_transaction,\n> > > > +\t\t\t\t struct strbuf *err)\n> > > > +{\n> > > > +\tint i;\n> > > > +\n> > > > +\tfor (i = 0; i < transaction->nr; i++) {\n> > > > +\t\tint last;\n> > > > +\t\tstruct ref_update *update = transaction\n> > > > ->updates[i];\n> > > > +\n> > > > +\t\tif (ref_type(update->refname) ==\n> > > > REF_TYPE_NORMAL)\n> > > > +\t\t\tcontinue;\n> > > > +\n> > > > +\t\tlast = --transaction->nr;\n> > > > +\t\ttransaction->updates[i] = transaction\n> > > > ->updates[last];\n> > > > +\t\tadd_update_obj(files_transaction, update);\n> > > > +\t}\n> > > > +\n> > > > +\treturn 0;\n> > > > +}\n> > > > +\n> > > \n> > > I think this function is incorrect. The update that was\n> > > previously at\n> > > transaction->updates[i] never gets checked for abnormality. \n> > \n> > Yes it does; that's the \"update\" variable that we just checked.\n> \n> Sorry, I meant to say \"the update that was previously at\n> `transaction->updates[transaction->nr - 1]` never gets checked for\n> abnormality\". Because it gets moved to `transaction->updates[i]`,\n> then\n> `i` is incremented without checking it.\n\nOh, right. \n"}]}