{"thread":{"id":"40926","subject":"[PATCH 00/16] LMDB refs backend atop pre-vtable","startedAt":"2015-12-03T00:35:05Z","lastAt":"2016-01-08T16:05:53Z","messageCount":73,"participants":["David Turner","Junio C Hamano","Duy Nguyen","Jeff King","Jonathan Nieder","Howard Chu","Michael Haggerty","Eric Sunshine","David Aguilar","Ronnie Sahlberg","Shawn Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":16},"messages":[{"id":"273946","messageId":"1449102921-7707-1-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":null,"subject":"[PATCH 00/16] LMDB refs backend atop pre-vtable","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:05Z","receivedAt":"2015-12-03T00:35:05Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"I'm starting the patchset numbering over from 1 here, because this\nversion of the patchset is a subset of the last version.\n\nThis version of the patch set applies on top of\ndt/refs-backend-pre-vtable. This required moving a bunch of stuff\nthat was in refs.h in previous versions into refs/refs-internal.h.\n\nSince the last patchset, I added support for symlink HEAD refs, and\nbroke out the initdb stuff into a separate commit.\n\nI also rearranged the order of the backend functions to make the vtable\neasier to read.\n\nI removed for_each_reftype_fullpath, which was at one point in next\nbut is not anymore. And I did a bit more code cleanup/rearrangement\non the LMDB stuff: I removed memory leaks, improved style, and just\ngenerally spruced things up.\n\nAs usual, the normal tests and the same set of hacked tests pass.\n\nI've read over each of these patches a few times, but I've probably\nstill managed to miss things. I look forward to your review.\n\nDavid Turner (13):\n refs: add do_for_each_per_worktree_ref\n refs: add methods for reflog\n refs: add method for initial ref transaction commit\n refs: add method for delete_refs\n refs: add methods to init refs backend and db\n refs: add method to rename refs\n refs: make lock generic\n refs: move duplicate check to common code\n refs: always handle non-normal refs in files backend\n init: allow alternate backends to be set for new repos\n refs: allow ref backend to be set for clone\n refs: add LMDB refs backend\n refs: tests for lmdb backend\n\nRonnie Sahlberg (3):\n refs: add a backend method structure with transaction functions\n refs: add methods for misc ref operations\n refs: add methods for the ref iterators\n\n .gitignore | 1 +\n Documentation/config.txt | 7 +\n Documentation/git-clone.txt | 4 +\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 7 +-\n Documentation/technical/refs-lmdb-backend.txt | 50 +\n Documentation/technical/repository-version.txt | 5 +\n Makefile | 12 +\n builtin/clone.c | 27 +-\n builtin/init-db.c | 35 +-\n builtin/submodule--helper.c | 5 +-\n cache.h | 10 +\n config.c | 34 +\n configure.ac | 33 +\n contrib/workdir/git-new-workdir | 3 +\n environment.c | 1 +\n path.c | 32 +-\n refs.c | 406 ++++-\n refs.h | 17 +\n refs/files-backend.c | 257 +--\n refs/lmdb-backend.c | 2054 ++++++++++++++++++++++++\n refs/refs-internal.h | 118 +-\n setup.c | 32 +-\n t/t1460-refs-lmdb-backend.sh | 1109 +++++++++++++\n t/t1470-refs-lmdb-backend-reflog.sh | 359 +++++\n t/test-lib.sh | 1 +\n test-refs-lmdb-backend.c | 68 +\n 27 files changed, 4553 insertions(+), 136 deletions(-)\n create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n create mode 100644 refs/lmdb-backend.c\n create mode 100755 t/t1460-refs-lmdb-backend.sh\n create mode 100755 t/t1470-refs-lmdb-backend-reflog.sh\n create mode 100644 test-refs-lmdb-backend.c\n\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273947","messageId":"1449102921-7707-2-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 01/16] refs: add a backend method structure with transaction functions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:06Z","receivedAt":"2015-12-03T00:35:06Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd a ref structure for backend methods. Start by adding a method pointer\nfor the transaction commit function.\n\nAdd a function set_refs_backend to switch between backends. The files\nbased backend is the default.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 32 ++++++++++++++++++++++++++++++++\n refs.h | 2 ++\n refs/files-backend.c | 10 ++++++++--\n refs/refs-internal.h | 10 ++++++++++\n 4 files changed, 52 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 0f7628d..babba8a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -10,6 +10,31 @@\n #include \"tag.h\"\n \n /*\n+ * We always have a files backend and it is the default.\n+ */\n+extern struct ref_be refs_be_files;\n+struct ref_be *the_refs_backend = &refs_be_files;\n+/*\n+ * List of all available backends\n+ */\n+struct ref_be *refs_backends = &refs_be_files;\n+\n+/*\n+ * This function is used to switch to an alternate backend.\n+ */\n+int set_refs_backend(const char *name)\n+{\n+\tstruct ref_be *be;\n+\n+\tfor (be = refs_backends; be; be = be->next)\n+\t\tif (!strcmp(be->name, name)) {\n+\t\t\tthe_refs_backend = be;\n+\t\t\treturn 0;\n+\t\t}\n+\treturn 1;\n+}\n+\n+/*\n * How to handle various characters in refnames:\n * 0: An acceptable character for refs\n * 1: End-of-component\n@@ -1082,3 +1107,10 @@ int rename_ref_available(const char *oldname, const char *newname)\n \tstrbuf_release(&err);\n \treturn ret;\n }\n+\n+/* backend functions */\n+int ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_commit(transaction, err);\n+}\ndiff --git a/refs.h b/refs.h\nindex 7a04077..4e5477d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -508,4 +508,6 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n+int set_refs_backend(const char *name);\n+\n #endif /* REFS_H */\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4db3e36..be34772 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3123,8 +3123,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3510,3 +3510,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \tunlock_ref(lock);\n \treturn -1;\n }\n+\n+struct ref_be refs_be_files = {\n+\tNULL,\n+\t\"files\",\n+\tfiles_transaction_commit,\n+};\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex c7dded3..f2c74f3 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -197,4 +197,14 @@ const char *find_descendant_ref(const char *dirname,\n \n int rename_ref_available(const char *oldname, const char *newname);\n \n+/* refs backends */\n+typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err);\n+\n+struct ref_be {\n+\tstruct ref_be *next;\n+\tconst char *name;\n+\tref_transaction_commit_fn *transaction_commit;\n+};\n+\n #endif /* REFS_REFS_INTERNAL_H */\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273961","messageId":"1449102921-7707-3-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 02/16] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:07Z","receivedAt":"2015-12-03T00:35:07Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd ref backend methods for:\nresolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,\ncreate_symref, resolve_gitlink_ref.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n builtin/init-db.c | 1 +\n cache.h | 7 +++++++\n refs.c | 36 ++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 34 +++++++++++++++++++++++-----------\n refs/refs-internal.h | 23 +++++++++++++++++++++++\n 5 files changed, 90 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 07229d6..26e1cc3 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -8,6 +8,7 @@\n #include \"builtin.h\"\n #include \"exec_cmd.h\"\n #include \"parse-options.h\"\n+#include \"refs.h\"\n \n #ifndef DEFAULT_GIT_TEMPLATE_DIR\n #define DEFAULT_GIT_TEMPLATE_DIR \"/usr/share/git-core/templates\"\ndiff --git a/cache.h b/cache.h\nindex 51c35c3..707455a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1111,6 +1111,13 @@ extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as s\n extern int interpret_branch_name(const char *str, int len, struct strbuf *);\n extern int get_sha1_mb(const char *str, unsigned char *sha1);\n \n+/*\n+ * Return true iff abbrev_name is a possible abbreviation for\n+ * full_name according to the rules defined by ref_rev_parse_rules in\n+ * refs.c.\n+ */\n+extern int refname_match(const char *abbrev_name, const char *full_name);\n+\n extern int validate_headref(const char *ref);\n \n extern int base_name_compare(const char *name1, int len1, int mode1, const char *name2, int len2, int mode2);\ndiff --git a/refs.c b/refs.c\nindex babba8a..9562325 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1114,3 +1114,39 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n {\n \treturn the_refs_backend->transaction_commit(transaction, err);\n }\n+\n+const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n+\t\t\t unsigned char *sha1, int *flags)\n+{\n+\treturn the_refs_backend->resolve_ref_unsafe(ref, resolve_flags, sha1,\n+\t\t\t\t\t\t flags);\n+}\n+\n+int verify_refname_available(const char *refname, struct string_list *extra,\n+\t\t\t struct string_list *skip, struct strbuf *err)\n+{\n+\treturn the_refs_backend->verify_refname_available(refname, extra, skip, err);\n+}\n+\n+int pack_refs(unsigned int flags)\n+{\n+\treturn the_refs_backend->pack_refs(flags);\n+}\n+\n+int peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\treturn the_refs_backend->peel_ref(refname, sha1);\n+}\n+\n+int create_symref(const char *ref_target, const char *refs_heads_master,\n+\t\t const char *logmsg)\n+{\n+\treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n+\t\t\t\t\t logmsg);\n+}\n+\n+int resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\tunsigned char *sha1)\n+{\n+\treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex be34772..25fba43 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1333,7 +1333,8 @@ static int resolve_gitlink_ref_recursive(struct ref_cache *refs,\n \treturn resolve_gitlink_ref_recursive(refs, p, sha1, recursion+1);\n }\n \n-int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *sha1)\n+static int files_resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n {\n \tint len = strlen(path), retval;\n \tchar *submodule;\n@@ -1565,8 +1566,10 @@ static const char *resolve_ref_1(const char *refname,\n \t}\n }\n \n-const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n-\t\t\t unsigned char *sha1, int *flags)\n+static const char *files_resolve_ref_unsafe(const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n {\n \tstatic struct strbuf sb_refname = STRBUF_INIT;\n \tstruct strbuf sb_contents = STRBUF_INIT;\n@@ -1615,7 +1618,7 @@ static enum peel_status peel_entry(struct ref_entry *entry, int repeel)\n \treturn status;\n }\n \n-int peel_ref(const char *refname, unsigned char *sha1)\n+static int files_peel_ref(const char *refname, unsigned char *sha1)\n {\n \tint flag;\n \tunsigned char base[20];\n@@ -2246,7 +2249,7 @@ static void prune_refs(struct ref_to_prune *r)\n \t}\n }\n \n-int pack_refs(unsigned int flags)\n+static int files_pack_refs(unsigned int flags)\n {\n \tstruct pack_refs_cb_data cbdata;\n \n@@ -2437,10 +2440,10 @@ out:\n \treturn ret;\n }\n \n-int verify_refname_available(const char *newname,\n-\t\t\t struct string_list *extras,\n-\t\t\t struct string_list *skip,\n-\t\t\t struct strbuf *err)\n+static int files_verify_refname_available(const char *newname,\n+\t\t\t\t\t struct string_list *extras,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tstruct ref_dir *packed_refs = get_packed_refs(&ref_cache);\n \tstruct ref_dir *loose_refs = get_loose_refs(&ref_cache);\n@@ -2811,8 +2814,9 @@ static int commit_ref_update(struct ref_lock *lock,\n \treturn 0;\n }\n \n-int create_symref(const char *ref_target, const char *refs_heads_master,\n-\t\t const char *logmsg)\n+static int files_create_symref(const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg)\n {\n \tchar *lockpath = NULL;\n \tchar ref[1000];\n@@ -3515,4 +3519,12 @@ struct ref_be refs_be_files = {\n \tNULL,\n \t\"files\",\n \tfiles_transaction_commit,\n+\n+\tfiles_pack_refs,\n+\tfiles_peel_ref,\n+\tfiles_create_symref,\n+\n+\tfiles_resolve_ref_unsafe,\n+\tfiles_verify_refname_available,\n+\tfiles_resolve_gitlink_ref,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f2c74f3..236bce9 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -201,10 +201,33 @@ int rename_ref_available(const char *oldname, const char *newname);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n+/* misc methods */\n+typedef int pack_refs_fn(unsigned int flags);\n+typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n+typedef int create_symref_fn(const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg);\n+\n+/* resolution methods */\n+typedef const char *resolve_ref_unsafe_fn(const char *ref,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags);\n+typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);\n+typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1);\n+\n struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n+\n+\tpack_refs_fn *pack_refs;\n+\tpeel_ref_fn *peel_ref;\n+\tcreate_symref_fn *create_symref;\n+\n+\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n+\tverify_refname_available_fn *verify_refname_available;\n+\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n };\n \n #endif /* REFS_REFS_INTERNAL_H */\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273960","messageId":"1449102921-7707-4-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 03/16] refs: add methods for the ref iterators","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:08Z","receivedAt":"2015-12-03T00:35:08Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n 3 files changed, 112 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9562325..b9b0244 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1150,3 +1150,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n {\n \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n }\n+\n+int head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref(fn, cb_data);\n+}\n+\n+int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref(fn, cb_data);\n+}\n+\n+int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in(prefix, fn, cb_data);\n+}\n+\n+int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data,\n+\t\t\tunsigned int broken)\n+{\n+\treturn the_refs_backend->for_each_fullref_in(prefix, fn, cb_data,\n+\t\t\t\t\t\t broken);\n+}\n+\n+int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n+\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in_submodule(submodule, prefix,\n+\t\t\t\t\t\t\t fn, cb_data);\n+}\n+\n+int for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_rawref(fn, cb_data);\n+}\n+\n+int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_namespaced_ref(fn, cb_data);\n+}\n+\n+int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 25fba43..d4bd6cf 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1756,32 +1756,36 @@ static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n \treturn 0;\n }\n \n-int head_ref(each_ref_fn fn, void *cb_data)\n+static int files_head_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_head_ref(NULL, fn, cb_data);\n }\n \n-int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_head_ref(submodule, fn, cb_data);\n }\n \n-int for_each_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t\tvoid *cb_data)\n {\n \treturn do_for_each_ref(get_ref_cache(submodule), \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, prefix, fn, strlen(prefix), 0, cb_data);\n }\n \n-int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)\n+static int files_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n {\n \tunsigned int flag = 0;\n \n@@ -1790,19 +1794,21 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig\n \treturn do_for_each_ref(&ref_cache, prefix, fn, 0, flag, cb_data);\n }\n \n-int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n-\t\teach_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn,\n+\t\t\t strlen(prefix), 0, cb_data);\n }\n \n-int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, git_replace_ref_base, fn,\n \t\t\t strlen(git_replace_ref_base), 0, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n@@ -1812,7 +1818,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \treturn ret;\n }\n \n-int for_each_rawref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_rawref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0,\n \t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n@@ -3527,4 +3533,15 @@ struct ref_be refs_be_files = {\n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\n \tfiles_resolve_gitlink_ref,\n+\n+\tfiles_head_ref,\n+\tfiles_head_ref_submodule,\n+\tfiles_for_each_ref,\n+\tfiles_for_each_ref_submodule,\n+\tfiles_for_each_ref_in,\n+\tfiles_for_each_fullref_in,\n+\tfiles_for_each_ref_in_submodule,\n+\tfiles_for_each_rawref,\n+\tfiles_for_each_namespaced_ref,\n+\tfiles_for_each_replace_ref,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 236bce9..ad683df 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -216,6 +216,24 @@ typedef int verify_refname_available_fn(const char *refname, struct string_list\n typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n \n+/* iteration methods */\n+typedef int head_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int head_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_ref_submodule_fn(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_ref_in_fn(const char *prefix, each_ref_fn fn,\n+\t\t\t void *cb_data);\n+typedef int for_each_fullref_in_fn(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken);\n+typedef int for_each_ref_in_submodule_fn(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data);\n+typedef int for_each_rawref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_namespaced_ref_fn(each_ref_fn fn, void *cb_data);\n+typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n+\n struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n@@ -228,6 +246,17 @@ struct ref_be {\n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n \tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n+\n+\thead_ref_fn *head_ref;\n+\thead_ref_submodule_fn *head_ref_submodule;\n+\tfor_each_ref_fn *for_each_ref;\n+\tfor_each_ref_submodule_fn *for_each_ref_submodule;\n+\tfor_each_ref_in_fn *for_each_ref_in;\n+\tfor_each_fullref_in_fn *for_each_fullref_in;\n+\tfor_each_ref_in_submodule_fn *for_each_ref_in_submodule;\n+\tfor_each_rawref_fn *for_each_rawref;\n+\tfor_each_namespaced_ref_fn *for_each_namespaced_ref;\n+\tfor_each_replace_ref_fn *for_each_replace_ref;\n };\n \n #endif /* REFS_REFS_INTERNAL_H */\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273949","messageId":"1449102921-7707-5-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 04/16] refs: add do_for_each_per_worktree_ref","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:09Z","receivedAt":"2015-12-03T00:35:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might still use files to store per-worktree\nrefs. So the files backend's ref-loading infrastructure should be\navailable to those backends, just for use on per-worktree refs. Add\ndo_for_each_per_worktree_ref, which iterates over per-worktree refs.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 15 ++++++++++++---\n refs/refs-internal.h | 10 ++++++++++\n 2 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d4bd6cf..bde4892 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -518,9 +518,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n \tdir->sorted = dir->nr = i;\n }\n \n-/* Include broken references in a do_for_each_ref*() iteration: */\n-#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n-\n /*\n * Return true iff the reference described by entry can be resolved to\n * an object in the database. Emit a warning if the referred-to\n@@ -568,6 +565,10 @@ static int do_one_ref(struct ref_entry *entry, void *cb_data)\n \tstruct ref_entry *old_current_ref;\n \tint retval;\n \n+\tif (data->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t ref_type(entry->name) != REF_TYPE_PER_WORKTREE)\n+\t\treturn 0;\n+\n \tif (!starts_with(entry->name, data->base))\n \t\treturn 0;\n \n@@ -1738,6 +1739,14 @@ static int do_for_each_ref(struct ref_cache *refs, const char *base,\n \treturn do_for_each_entry(refs, base, do_one_ref, &data);\n }\n \n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(get_ref_cache(submodule), base, fn, trim,\n+\t\t\t flags | DO_FOR_EACH_PER_WORKTREE_ONLY, cb_data);\n+}\n+\n static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n \tstruct object_id oid;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex ad683df..433d0fe 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -42,6 +42,16 @@\n * value to ref_update::flags\n */\n \n+/* Include broken references in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n+\n+/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n+\n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data);\n+\n /*\n * Return true iff refname is minimally safe. \"Safe\" here means that\n * deleting a loose reference by this name will not do any damage, for\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273962","messageId":"1449102921-7707-6-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 05/16] refs: add methods for reflog","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:10Z","receivedAt":"2015-12-03T00:35:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In the file-based backend, the reflog piggybacks on the ref lock.\nSince other backends won't have the same sort of ref lock, ref backends\nmust also handle reflogs.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n refs/files-backend.c | 36 ++++++++++++++++++++++++------------\n refs/refs-internal.h | 27 +++++++++++++++++++++++++++\n 3 files changed, 97 insertions(+), 12 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex b9b0244..a8ed77d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1204,3 +1204,49 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n }\n+\n+int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,\n+\t\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent_reverse(refname, fn,\n+\t\t\t\t\t\t\t cb_data);\n+}\n+\n+int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,\n+\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent(refname, fn, cb_data);\n+}\n+\n+int for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog(fn, cb_data);\n+}\n+\n+int reflog_exists(const char *refname)\n+{\n+\treturn the_refs_backend->reflog_exists(refname);\n+}\n+\n+int safe_create_reflog(const char *refname, int force_create,\n+\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->create_reflog(refname, force_create, err);\n+}\n+\n+int delete_reflog(const char *refname)\n+{\n+\treturn the_refs_backend->delete_reflog(refname);\n+}\n+\n+int reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\treturn the_refs_backend->reflog_expire(refname, sha1, flags,\n+\t\t\t\t\t prepare_fn, should_prune_fn,\n+\t\t\t\t\t cleanup_fn, policy_cb_data);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bde4892..1f76e34 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2643,7 +2643,8 @@ static int log_ref_setup(const char *refname, struct strbuf *logfile, struct str\n }\n \n \n-int safe_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+static int files_create_reflog(const char *refname, int force_create,\n+\t\t\t struct strbuf *err)\n {\n \tint ret;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -2899,7 +2900,7 @@ static int files_create_symref(const char *ref_target,\n \treturn 0;\n }\n \n-int reflog_exists(const char *refname)\n+static int files_reflog_exists(const char *refname)\n {\n \tstruct stat st;\n \n@@ -2907,7 +2908,7 @@ int reflog_exists(const char *refname)\n \t\tS_ISREG(st.st_mode);\n }\n \n-int delete_reflog(const char *refname)\n+static int files_delete_reflog(const char *refname)\n {\n \treturn remove_path(git_path(\"logs/%s\", refname));\n }\n@@ -2951,7 +2952,9 @@ static char *find_beginning_of_line(char *bob, char *scan)\n \treturn scan;\n }\n \n-int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tFILE *logfp;\n@@ -3053,7 +3056,8 @@ int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void\n \treturn ret;\n }\n \n-int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn, void *cb_data)\n {\n \tFILE *logfp;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -3115,7 +3119,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data\n \treturn retval;\n }\n \n-int for_each_reflog(each_ref_fn fn, void *cb_data)\n+static int files_for_each_reflog(each_ref_fn fn, void *cb_data)\n {\n \tint retval;\n \tstruct strbuf name;\n@@ -3425,12 +3429,12 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \treturn 0;\n }\n \n-int reflog_expire(const char *refname, const unsigned char *sha1,\n-\t\t unsigned int flags,\n-\t\t reflog_expiry_prepare_fn prepare_fn,\n-\t\t reflog_expiry_should_prune_fn should_prune_fn,\n-\t\t reflog_expiry_cleanup_fn cleanup_fn,\n-\t\t void *policy_cb_data)\n+static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n {\n \tstatic struct lock_file reflog_lock;\n \tstruct expire_reflog_cb cb;\n@@ -3535,6 +3539,14 @@ struct ref_be refs_be_files = {\n \t\"files\",\n \tfiles_transaction_commit,\n \n+\tfiles_for_each_reflog_ent,\n+\tfiles_for_each_reflog_ent_reverse,\n+\tfiles_for_each_reflog,\n+\tfiles_reflog_exists,\n+\tfiles_create_reflog,\n+\tfiles_delete_reflog,\n+\tfiles_reflog_expire,\n+\n \tfiles_pack_refs,\n \tfiles_peel_ref,\n \tfiles_create_symref,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 433d0fe..798dee9 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -211,6 +211,25 @@ int rename_ref_available(const char *oldname, const char *newname);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n+/* reflog functions */\n+typedef int for_each_reflog_ent_fn(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int for_each_reflog_ent_reverse_fn(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data);\n+typedef int for_each_reflog_fn(each_ref_fn fn, void *cb_data);\n+typedef int reflog_exists_fn(const char *refname);\n+typedef int create_reflog_fn(const char *refname, int force_create,\n+\t\t\t struct strbuf *err);\n+typedef int delete_reflog_fn(const char *refname);\n+typedef int reflog_expire_fn(const char *refname, const unsigned char *sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data);\n+\n /* misc methods */\n typedef int pack_refs_fn(unsigned int flags);\n typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n@@ -249,6 +268,14 @@ struct ref_be {\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n \n+\tfor_each_reflog_ent_fn *for_each_reflog_ent;\n+\tfor_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;\n+\tfor_each_reflog_fn *for_each_reflog;\n+\treflog_exists_fn *reflog_exists;\n+\tcreate_reflog_fn *create_reflog;\n+\tdelete_reflog_fn *delete_reflog;\n+\treflog_expire_fn *reflog_expire;\n+\n \tpack_refs_fn *pack_refs;\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273948","messageId":"1449102921-7707-7-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 06/16] refs: add method for initial ref transaction commit","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:11Z","receivedAt":"2015-12-03T00:35:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 6 ++++++\n refs/files-backend.c | 5 +++--\n refs/refs-internal.h | 1 +\n 3 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex a8ed77d..e50516c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1250,3 +1250,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t\t\t prepare_fn, should_prune_fn,\n \t\t\t\t\t cleanup_fn, policy_cb_data);\n }\n+\n+int initial_ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->initial_transaction_commit(transaction, err);\n+}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1f76e34..44ad632 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3313,8 +3313,8 @@ static int ref_present(const char *refname,\n \treturn string_list_has_string(affected_refnames, refname);\n }\n \n-int initial_ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t\t struct strbuf *err)\n+static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3538,6 +3538,7 @@ struct ref_be refs_be_files = {\n \tNULL,\n \t\"files\",\n \tfiles_transaction_commit,\n+\tfiles_initial_transaction_commit,\n \n \tfiles_for_each_reflog_ent,\n \tfiles_for_each_reflog_ent_reverse,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 798dee9..74bd44b 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -267,6 +267,7 @@ struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n \tref_transaction_commit_fn *transaction_commit;\n+\tref_transaction_commit_fn *initial_transaction_commit;\n \n \tfor_each_reflog_ent_fn *for_each_reflog_ent;\n \tfor_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273950","messageId":"1449102921-7707-8-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 07/16] refs: add method for delete_refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:12Z","receivedAt":"2015-12-03T00:35:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In the file-based backend, delete_refs has some special optimization\nto deal with packed refs. In other backends, we might be able to make\nref deletion faster by putting all deletions into a single\ntransaction. So we need a special backend function for this.\n\nSigned-off-by: David Turner \n---\n refs.c | 5 +++++\n refs/files-backend.c | 3 ++-\n refs/refs-internal.h | 2 ++\n 3 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex e50516c..9a2fed7 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1115,6 +1115,11 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn the_refs_backend->transaction_commit(transaction, err);\n }\n \n+int delete_refs(struct string_list *refnames)\n+{\n+\treturn the_refs_backend->delete_refs(refnames);\n+}\n+\n const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags)\n {\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 44ad632..e769242 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2356,7 +2356,7 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n \treturn 0;\n }\n \n-int delete_refs(struct string_list *refnames)\n+static int files_delete_refs(struct string_list *refnames)\n {\n \tstruct strbuf err = STRBUF_INIT;\n \tint i, result = 0;\n@@ -3551,6 +3551,7 @@ struct ref_be refs_be_files = {\n \tfiles_pack_refs,\n \tfiles_peel_ref,\n \tfiles_create_symref,\n+\tfiles_delete_refs,\n \n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74bd44b..478ad54 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -236,6 +236,7 @@ typedef int peel_ref_fn(const char *refname, unsigned char *sha1);\n typedef int create_symref_fn(const char *ref_target,\n \t\t\t const char *refs_heads_master,\n \t\t\t const char *logmsg);\n+typedef int delete_refs_fn(struct string_list *refnames);\n \n /* resolution methods */\n typedef const char *resolve_ref_unsafe_fn(const char *ref,\n@@ -280,6 +281,7 @@ struct ref_be {\n \tpack_refs_fn *pack_refs;\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n+\tdelete_refs_fn *delete_refs;\n \n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273959","messageId":"1449102921-7707-9-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 08/16] refs: add methods to init refs backend and db","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:13Z","receivedAt":"2015-12-03T00:35:13Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might not need the refs/heads directory and so\non, so we make ref db initialization part of the backend. We also\nmight need to initialize ref backends themselves, so we'll add a\nmethod for that as well.\n\nSigned-off-by: David Turner \n---\n builtin/init-db.c | 14 ++++----------\n refs.c | 8 +++++++-\n refs.h | 4 +++-\n refs/files-backend.c | 23 +++++++++++++++++++++++\n refs/refs-internal.h | 4 ++++\n 5 files changed, 41 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 26e1cc3..4771e7e 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -178,13 +178,7 @@ static int create_default_files(const char *template_path)\n \tchar junk[2];\n \tint reinit;\n \tint filemode;\n-\n-\t/*\n-\t * Create .git/refs/{heads,tags}\n-\t */\n-\tsafe_create_dir(git_path_buf(&buf, \"refs\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/heads\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/tags\"), 1);\n+\tstruct strbuf err = STRBUF_INIT;\n \n \t/* Just look for `init.templatedir` */\n \tgit_config(git_init_db_config, NULL);\n@@ -208,11 +202,11 @@ static int create_default_files(const char *template_path)\n \t */\n \tif (shared_repository) {\n \t\tadjust_shared_perm(get_git_dir());\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/heads\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/tags\"));\n \t}\n \n+\tif (refs_init_db(&err, shared_repository))\n+\t\tdie(\"failed to set up refs db: %s\", err.buf);\n+\n \t/*\n \t * Create the default symlink from \".git/HEAD\" to the \"master\"\n \t * branch, if it does not exist yet.\ndiff --git a/refs.c b/refs.c\nindex 9a2fed7..bdeb276 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -22,13 +22,14 @@ struct ref_be *refs_backends = &refs_be_files;\n /*\n * This function is used to switch to an alternate backend.\n */\n-int set_refs_backend(const char *name)\n+int set_refs_backend(const char *name, void *data)\n {\n \tstruct ref_be *be;\n \n \tfor (be = refs_backends; be; be = be->next)\n \t\tif (!strcmp(be->name, name)) {\n \t\t\tthe_refs_backend = be;\n+\t\t\tbe->init_backend(data);\n \t\t\treturn 0;\n \t\t}\n \treturn 1;\n@@ -1109,6 +1110,11 @@ int rename_ref_available(const char *oldname, const char *newname)\n }\n \n /* backend functions */\n+int refs_init_db(struct strbuf *err, int shared)\n+{\n+\treturn the_refs_backend->init_db(err, shared);\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\ndiff --git a/refs.h b/refs.h\nindex 4e5477d..c211b9e 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -66,6 +66,8 @@ extern int ref_exists(const char *refname);\n \n extern int is_branch(const char *refname);\n \n+extern int refs_init_db(struct strbuf *err, int shared);\n+\n /*\n * If refname is a non-symbolic reference that refers to a tag object,\n * and the tag can be (recursively) dereferenced to a non-tag object,\n@@ -508,6 +510,6 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n-int set_refs_backend(const char *name);\n+int set_refs_backend(const char *name, void *data);\n \n #endif /* REFS_H */\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e769242..6600c02 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3313,6 +3313,11 @@ static int ref_present(const char *refname,\n \treturn string_list_has_string(affected_refnames, refname);\n }\n \n+void files_init_backend(void *data)\n+{\n+\t/* do nothing */\n+}\n+\n static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t struct strbuf *err)\n {\n@@ -3534,9 +3539,27 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \treturn -1;\n }\n \n+static int files_init_db(struct strbuf *err, int shared)\n+{\n+\t/*\n+\t * Create .git/refs/{heads,tags}\n+\t */\n+\tsafe_create_dir(git_path(\"refs\"), 1);\n+\tsafe_create_dir(git_path(\"refs/heads\"), 1);\n+\tsafe_create_dir(git_path(\"refs/tags\"), 1);\n+\tif (shared) {\n+\t\tadjust_shared_perm(git_path(\"refs\"));\n+\t\tadjust_shared_perm(git_path(\"refs/heads\"));\n+\t\tadjust_shared_perm(git_path(\"refs/tags\"));\n+\t}\n+\treturn 0;\n+}\n+\n struct ref_be refs_be_files = {\n \tNULL,\n \t\"files\",\n+\tfiles_init_backend,\n+\tfiles_init_db,\n \tfiles_transaction_commit,\n \tfiles_initial_transaction_commit,\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 478ad54..85a0b91 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -208,6 +208,8 @@ const char *find_descendant_ref(const char *dirname,\n int rename_ref_available(const char *oldname, const char *newname);\n \n /* refs backends */\n+typedef void ref_backend_init_fn(void *data);\n+typedef int ref_backend_init_db_fn(struct strbuf *err, int shared);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n \n@@ -267,6 +269,8 @@ typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n+\tref_backend_init_fn *init_backend;\n+\tref_backend_init_db_fn *init_db;\n \tref_transaction_commit_fn *transaction_commit;\n \tref_transaction_commit_fn *initial_transaction_commit;\n \n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273957","messageId":"1449102921-7707-10-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 09/16] refs: add method to rename refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:14Z","receivedAt":"2015-12-03T00:35:14Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: David Turner \n---\n refs.c | 37 +++++++++++++++++++++----------------\n refs/files-backend.c | 4 +++-\n refs/refs-internal.h | 9 +++++++++\n 3 files changed, 33 insertions(+), 17 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex bdeb276..1b79630 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1093,22 +1093,6 @@ const char *find_descendant_ref(const char *dirname,\n \treturn NULL;\n }\n \n-int rename_ref_available(const char *oldname, const char *newname)\n-{\n-\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n-\tstruct strbuf err = STRBUF_INIT;\n-\tint ret;\n-\n-\tstring_list_insert(&skip, oldname);\n-\tret = !verify_refname_available(newname, NULL, &skip, &err);\n-\tif (!ret)\n-\t\terror(\"%s\", err.buf);\n-\n-\tstring_list_clear(&skip, 0);\n-\tstrbuf_release(&err);\n-\treturn ret;\n-}\n-\n /* backend functions */\n int refs_init_db(struct strbuf *err, int shared)\n {\n@@ -1126,6 +1110,11 @@ int delete_refs(struct string_list *refnames)\n \treturn the_refs_backend->delete_refs(refnames);\n }\n \n+int rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\treturn the_refs_backend->rename_ref(oldref, newref, logmsg);\n+}\n+\n const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags)\n {\n@@ -1139,6 +1128,22 @@ int verify_refname_available(const char *refname, struct string_list *extra,\n \treturn the_refs_backend->verify_refname_available(refname, extra, skip, err);\n }\n \n+int rename_ref_available(const char *oldname, const char *newname)\n+{\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint ret;\n+\n+\tstring_list_insert(&skip, oldname);\n+\tret = !verify_refname_available(newname, NULL, &skip, &err);\n+\tif (!ret)\n+\t\terror(\"%s\", err.buf);\n+\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&err);\n+\treturn ret;\n+}\n+\n int pack_refs(unsigned int flags)\n {\n \treturn the_refs_backend->pack_refs(flags);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6600c02..0af0818 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2478,7 +2478,8 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t const unsigned char *sha1, const char *logmsg,\n \t\t\t int flags, struct strbuf *err);\n \n-int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)\n+static int files_rename_ref(const char *oldrefname, const char *newrefname,\n+\t\t\t const char *logmsg)\n {\n \tunsigned char sha1[20], orig_sha1[20];\n \tint flag = 0, logmoved = 0;\n@@ -3575,6 +3576,7 @@ struct ref_be refs_be_files = {\n \tfiles_peel_ref,\n \tfiles_create_symref,\n \tfiles_delete_refs,\n+\tfiles_rename_ref,\n \n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 85a0b91..36a024f 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -53,6 +53,13 @@ int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n \t\t\t\t void *cb_data);\n \n /*\n+ * Check if the new name does not conflict with any existing refs\n+ * (other than possibly the old ref). Return 0 if the ref can be\n+ * renamed to the new name.\n+ */\n+int rename_ref_available(const char *oldname, const char *newname);\n+\n+/*\n * Return true iff refname is minimally safe. \"Safe\" here means that\n * deleting a loose reference by this name will not do any damage, for\n * example by causing a file that is not a reference to be deleted.\n@@ -247,6 +254,7 @@ typedef const char *resolve_ref_unsafe_fn(const char *ref,\n typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);\n typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n+typedef int rename_ref_fn(const char *oldref, const char *newref, const char *logmsg);\n \n /* iteration methods */\n typedef int head_ref_fn(each_ref_fn fn, void *cb_data);\n@@ -286,6 +294,7 @@ struct ref_be {\n \tpeel_ref_fn *peel_ref;\n \tcreate_symref_fn *create_symref;\n \tdelete_refs_fn *delete_refs;\n+\trename_ref_fn *rename_ref;\n \n \tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n \tverify_refname_available_fn *verify_refname_available;\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273958","messageId":"1449102921-7707-11-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 10/16] refs: make lock generic","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:15Z","receivedAt":"2015-12-03T00:35:15Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Instead of using a files-backend-specific struct ref_lock, the generic\nref_transaction struct should provide a void pointer that backends can use\nfor their own lock data.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 29 ++++++++++++++++-------------\n refs/refs-internal.h | 2 +-\n 2 files changed, 17 insertions(+), 14 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0af0818..22a6f24 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3184,11 +3184,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t */\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n+\t\tstruct ref_lock *lock;\n \n \t\tif ((update->flags & REF_HAVE_NEW) &&\n \t\t is_null_sha1(update->new_sha1))\n \t\t\tupdate->flags |= REF_DELETING;\n-\t\tupdate->lock = lock_ref_sha1_basic(\n+\t\tlock = lock_ref_sha1_basic(\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n@@ -3196,7 +3197,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n \t\t\t\terr);\n-\t\tif (!update->lock) {\n+\t\tupdate->backend_data = lock;\n+\t\tif (!lock) {\n \t\t\tchar *reason;\n \n \t\t\tret = (errno == ENOTDIR)\n@@ -3214,12 +3216,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t\t\t (update->flags & REF_NODEREF));\n \n \t\t\tif (!overwriting_symref &&\n-\t\t\t !hashcmp(update->lock->old_oid.hash, update->new_sha1)) {\n+\t\t\t !hashcmp(lock->old_oid.hash, update->new_sha1)) {\n \t\t\t\t/*\n \t\t\t\t * The reference already has the desired\n \t\t\t\t * value, so we don't need to write it.\n \t\t\t\t */\n-\t\t\t} else if (write_ref_to_lockfile(update->lock,\n+\t\t\t} else if (write_ref_to_lockfile(lock,\n \t\t\t\t\t\t\t update->new_sha1,\n \t\t\t\t\t\t\t err)) {\n \t\t\t\tchar *write_err = strbuf_detach(err, NULL);\n@@ -3228,7 +3230,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t * The lock was freed upon failure of\n \t\t\t\t * write_ref_to_lockfile():\n \t\t\t\t */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t\tstrbuf_addf(err,\n \t\t\t\t\t \"cannot update the ref '%s': %s\",\n \t\t\t\t\t update->refname, write_err);\n@@ -3244,7 +3246,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t * We didn't have to write anything to the lockfile.\n \t\t\t * Close it to free up the file descriptor:\n \t\t\t */\n-\t\t\tif (close_ref(update->lock)) {\n+\t\t\tif (close_ref(lock)) {\n \t\t\t\tstrbuf_addf(err, \"Couldn't close %s.lock\",\n \t\t\t\t\t update->refname);\n \t\t\t\tgoto cleanup;\n@@ -3257,16 +3259,16 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \n \t\tif (update->flags & REF_NEEDS_COMMIT) {\n-\t\t\tif (commit_ref_update(update->lock,\n+\t\t\tif (commit_ref_update(update->backend_data,\n \t\t\t\t\t update->new_sha1, update->msg,\n \t\t\t\t\t update->flags, err)) {\n \t\t\t\t/* freed by commit_ref_update(): */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\n \t\t\t} else {\n \t\t\t\t/* freed by commit_ref_update(): */\n-\t\t\t\tupdate->lock = NULL;\n+\t\t\t\tupdate->backend_data = NULL;\n \t\t\t}\n \t\t}\n \t}\n@@ -3274,16 +3276,17 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t/* Perform deletes now that updates are safely completed */\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n+\t\tstruct ref_lock *lock = update->backend_data;\n \n \t\tif (update->flags & REF_DELETING) {\n-\t\t\tif (delete_ref_loose(update->lock, update->type, err)) {\n+\t\t\tif (delete_ref_loose(lock, update->type, err)) {\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (!(update->flags & REF_ISPRUNING))\n \t\t\t\tstring_list_append(&refs_to_delete,\n-\t\t\t\t\t\t update->lock->ref_name);\n+\t\t\t\t\t\t lock->ref_name);\n \t\t}\n \t}\n \n@@ -3299,8 +3302,8 @@ cleanup:\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \n \tfor (i = 0; i < n; i++)\n-\t\tif (updates[i]->lock)\n-\t\t\tunlock_ref(updates[i]->lock);\n+\t\tif (updates[i]->backend_data)\n+\t\t\tunlock_ref(updates[i]->backend_data);\n \tstring_list_clear(&refs_to_delete, 0);\n \tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 36a024f..8322011 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -161,7 +161,7 @@ struct ref_update {\n \t * REF_DELETING, and REF_ISPRUNING:\n \t */\n \tunsigned int flags;\n-\tstruct ref_lock *lock;\n+\tvoid *backend_data;\n \tint type;\n \tchar *msg;\n \tconst char refname[FLEX_ARRAY];\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273951","messageId":"1449102921-7707-12-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 11/16] refs: move duplicate check to common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:16Z","receivedAt":"2015-12-03T00:35:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The check for duplicate refnames in a transaction is needed for\nall backends, so move it to the common code.\n\nref_transaction_commit_fn gains a new argument, the sorted\nstring_list of affected refnames.\n\nSigned-off-by: David Turner \n---\n refs.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n refs/files-backend.c | 57 ++++-------------------------------------\n refs/refs-internal.h | 1 +\n 3 files changed, 75 insertions(+), 54 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1b79630..808053f 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1093,6 +1093,37 @@ const char *find_descendant_ref(const char *dirname,\n \treturn NULL;\n }\n \n+/*\n+ * Return 1 if there are any duplicate refnames in the updates in\n+ * `transaction`, and fill in err with an appropriate error message.\n+ * Fill in `refnames` with the refnames from the transaction.\n+ */\n+\n+static int ref_update_reject_duplicates(struct ref_transaction *transaction,\n+\t\t\t\t\tstruct string_list *refnames,\n+\t\t\t\t\tstruct strbuf *err)\n+{\n+\tint i, n = transaction->nr;\n+\tstruct ref_update **updates;\n+\n+\tassert(err);\n+\n+\tupdates = transaction->updates;\n+\t/* Fail if a refname appears more than once in the transaction: */\n+\tfor (i = 0; i < n; i++)\n+\t\tstring_list_append(refnames, updates[i]->refname);\n+\tstring_list_sort(refnames);\n+\n+\tfor (i = 1; i < n; i++)\n+\t\tif (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Multiple updates for ref '%s' not allowed.\",\n+\t\t\t\t refnames->items[i].string);\n+\t\t\treturn 1;\n+\t\t}\n+\treturn 0;\n+}\n+\n /* backend functions */\n int refs_init_db(struct strbuf *err, int shared)\n {\n@@ -1102,7 +1133,29 @@ int refs_init_db(struct strbuf *err, int shared)\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n-\treturn the_refs_backend->transaction_commit(transaction, err);\n+\tint ret = -1;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\n+\tassert(err);\n+\n+\tif (transaction->state != REF_TRANSACTION_OPEN)\n+\t\tdie(\"BUG: commit called for transaction that is not open\");\n+\n+\tif (!transaction->nr) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\treturn 0;\n+\t}\n+\n+\tif (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = the_refs_backend->transaction_commit(transaction,\n+\t\t\t\t\t\t &affected_refnames, err);\n+done:\n+\tstring_list_clear(&affected_refnames, 0);\n+\treturn ret;\n }\n \n int delete_refs(struct string_list *refnames)\n@@ -1270,5 +1323,19 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n int initial_ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n {\n-\treturn the_refs_backend->initial_transaction_commit(transaction, err);\n+\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tint ret;\n+\n+\tif (ref_update_reject_duplicates(transaction,\n+\t\t\t\t\t &affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\tret = the_refs_backend->initial_transaction_commit(transaction,\n+\t\t\t\t\t\t\t &affected_refnames,\n+\t\t\t\t\t\t\t err);\n+done:\n+\tstring_list_clear(&affected_refnames, 0);\n+\treturn ret;\n }\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 22a6f24..59e2ec1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3130,24 +3130,8 @@ static int files_for_each_reflog(each_ref_fn fn, void *cb_data)\n \treturn retval;\n }\n \n-static int ref_update_reject_duplicates(struct string_list *refnames,\n-\t\t\t\t\tstruct strbuf *err)\n-{\n-\tint i, n = refnames->nr;\n-\n-\tassert(err);\n-\n-\tfor (i = 1; i < n; i++)\n-\t\tif (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {\n-\t\t\tstrbuf_addf(err,\n-\t\t\t\t \"Multiple updates for ref '%s' not allowed.\",\n-\t\t\t\t refnames->items[i].string);\n-\t\t\treturn 1;\n-\t\t}\n-\treturn 0;\n-}\n-\n static int files_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n@@ -3155,26 +3139,6 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \tstruct ref_update **updates = transaction->updates;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_NODUP;\n \tstruct string_list_item *ref_to_delete;\n-\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n-\n-\tassert(err);\n-\n-\tif (transaction->state != REF_TRANSACTION_OPEN)\n-\t\tdie(\"BUG: commit called for transaction that is not open\");\n-\n-\tif (!n) {\n-\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n-\t\treturn 0;\n-\t}\n-\n-\t/* Fail if a refname appears more than once in the transaction: */\n-\tfor (i = 0; i < n; i++)\n-\t\tstring_list_append(&affected_refnames, updates[i]->refname);\n-\tstring_list_sort(&affected_refnames);\n-\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n \n \t/*\n \t * Acquire all locks, verify old values if provided, check\n@@ -3193,7 +3157,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n-\t\t\t\t&affected_refnames, NULL,\n+\t\t\t\taffected_refnames, NULL,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n \t\t\t\terr);\n@@ -3305,7 +3269,6 @@ cleanup:\n \t\tif (updates[i]->backend_data)\n \t\t\tunlock_ref(updates[i]->backend_data);\n \tstring_list_clear(&refs_to_delete, 0);\n-\tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\n }\n \n@@ -3323,27 +3286,18 @@ void files_init_backend(void *data)\n }\n \n static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n-\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \n \tassert(err);\n \n \tif (transaction->state != REF_TRANSACTION_OPEN)\n \t\tdie(\"BUG: commit called for transaction that is not open\");\n \n-\t/* Fail if a refname appears more than once in the transaction: */\n-\tfor (i = 0; i < n; i++)\n-\t\tstring_list_append(&affected_refnames, updates[i]->refname);\n-\tstring_list_sort(&affected_refnames);\n-\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n-\n \t/*\n \t * It's really undefined to call this function in an active\n \t * repository or when there are existing references: we are\n@@ -3356,7 +3310,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \t * so here we really only check that none of the references\n \t * that we are creating already exists.\n \t */\n-\tif (for_each_rawref(ref_present, &affected_refnames))\n+\tif (for_each_rawref(ref_present, affected_refnames))\n \t\tdie(\"BUG: initial ref transaction called with existing refs\");\n \n \tfor (i = 0; i < n; i++) {\n@@ -3366,7 +3320,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \t\t !is_null_sha1(update->old_sha1))\n \t\t\tdie(\"BUG: initial ref transaction with old_sha1 set\");\n \t\tif (verify_refname_available(update->refname,\n-\t\t\t\t\t &affected_refnames, NULL,\n+\t\t\t\t\t affected_refnames, NULL,\n \t\t\t\t\t err)) {\n \t\t\tret = TRANSACTION_NAME_CONFLICT;\n \t\t\tgoto cleanup;\n@@ -3397,7 +3351,6 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,\n \n cleanup:\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n-\tstring_list_clear(&affected_refnames, 0);\n \treturn ret;\n }\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 8322011..9c17fdf 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -218,6 +218,7 @@ int rename_ref_available(const char *oldname, const char *newname);\n typedef void ref_backend_init_fn(void *data);\n typedef int ref_backend_init_db_fn(struct strbuf *err, int shared);\n typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n \t\t\t\t struct strbuf *err);\n \n /* reflog functions */\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273955","messageId":"1449102921-7707-13-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 12/16] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:17Z","receivedAt":"2015-12-03T00:35:17Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Always handle non-normal (per-worktree or pseudo) refs in the files\nbackend instead of alternate backends.\n\nSometimes a ref transaction will update both a per-worktree ref and a\nnormal ref. For instance, an ordinary commit might update\nrefs/heads/master and HEAD (or at least HEAD's reflog).\n\nWe handle three cases here:\n\n1. updates to normal refs continue to go through the chosen backend\n\n2. updates to non-normal refs with REF_NODEREF or to non-symbolic refs\nare moved to a separate files backend transaction.\n\n3. updates to symbolic refs are dereferenced to their base ref. The\nupdate to the base ref then goes through the ordinary backend, while\nthe files backend is directly called to update the symref's reflog.\n\nSigned-off-by: David Turner \n---\n refs.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 139 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 808053f..e48e43a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -9,6 +9,11 @@\n #include \"object.h\"\n #include \"tag.h\"\n \n+const char split_transaction_fail_warning[] =\n+\t\"A ref transaction was split across two refs backends. Part of the \"\n+\t\"transaction succeeded, but then the update to the per-worktree refs \"\n+\t\"failed. Your repository may be in an inconsistent state.\";\n+\n /*\n * We always have a files backend and it is the default.\n */\n@@ -784,6 +789,13 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \tfree(transaction);\n }\n \n+static void add_update_obj(struct ref_transaction *transaction,\n+\t\t\t struct ref_update *update)\n+{\n+\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n+\ttransaction->updates[transaction->nr++] = update;\n+}\n+\n static struct ref_update *add_update(struct ref_transaction *transaction,\n \t\t\t\t const char *refname)\n {\n@@ -791,8 +803,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \tstruct ref_update *update = xcalloc(1, sizeof(*update) + len);\n \n \tmemcpy((char *)update->refname, refname, len); /* includes NUL */\n-\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n-\ttransaction->updates[transaction->nr++] = update;\n+\tadd_update_obj(transaction, update);\n \treturn update;\n }\n \n@@ -1130,11 +1141,87 @@ int refs_init_db(struct strbuf *err, int shared)\n \treturn the_refs_backend->init_db(err, shared);\n }\n \n+/*\n+ * Special case for non-normal refs. For symbolic-refs when\n+ * REF_NODEREF is not turned on, we dereference them here and replace\n+ * updates to the symbolic refs with updates to the underlying ref.\n+ * Then we do our own reflogging for the symbolic ref.\n+ *\n+ * We move other non-normal ref updates with into a specially-created\n+ * files-backend transaction\n+ */\n+static int move_abnormal_ref_updates(struct ref_transaction *transaction,\n+\t\t\t\t struct ref_transaction *files_transaction,\n+\t\t\t\t struct string_list *symrefs)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tconst char *resolved;\n+\t\tint flags = 0;\n+\t\tunsigned char sha1[20];\n+\n+\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n+\t\t\tcontinue;\n+\n+\t\tresolved = resolve_ref_unsafe(update->refname, 0, sha1, &flags);\n+\n+\t\tif (update->flags & REF_NODEREF || !(flags & REF_ISSYMREF)) {\n+\t\t\tint last;\n+\n+\t\t\tadd_update_obj(files_transaction, update);\n+\t\t\t/*\n+\t\t\t * Replace this transaction with the\n+\t\t\t * last transaction, removing it from\n+\t\t\t * the list of backend transactions\n+\t\t\t */\n+\t\t\tlast = --transaction->nr;\n+\t\t\ttransaction->updates[i] = transaction->updates[last];\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (resolved) {\n+\t\t\tstruct ref_update *new_update;\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\tif (ref_type(resolved) != REF_TYPE_NORMAL)\n+\t\t\t\tdie(\"Non-normal symbolic ref `%s` points to non-normal ref `%s`\", update->refname, resolved);\n+\n+\t\t\tnew_update = xmalloc(sizeof(*new_update) +\n+\t\t\t\t\t strlen(resolved) + 1);\n+\t\t\tmemcpy(new_update, update, sizeof(*update));\n+\n+\t\t\tif (update->flags & REF_HAVE_OLD &&\n+\t\t\t hashcmp(sha1, update->old_sha1)) {\n+\t\t\t\t/* consistency check failed */\n+\t\t\t\tfree(new_update);\n+\t\t\t\treturn -1;\n+\t\t\t} else {\n+\t\t\t\thashcpy(update->old_sha1, sha1);\n+\t\t\t}\n+\n+\t\t\tstrcpy((char *)new_update->refname, resolved);\n+\t\t\ttransaction->updates[i] = new_update;\n+\n+\t\t\titem = string_list_append(symrefs, update->refname);\n+\t\t\titem->util = new_update;\n+\t\t\tfree(update);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n \tint ret = -1;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list symrefs = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tstruct ref_transaction *files_transaction = NULL;\n \n \tassert(err);\n \n@@ -1146,6 +1233,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\treturn 0;\n \t}\n \n+\tif (the_refs_backend != &refs_be_files) {\n+\t\tfiles_transaction = ref_transaction_begin(err);\n+\t\tif (!files_transaction)\n+\t\t\tdie(\"%s\", err->buf);\n+\n+\t\tret = move_abnormal_ref_updates(transaction, files_transaction,\n+\t\t\t\t\t\t&symrefs);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\t/* files backend commit */\n+\t\tif (ref_update_reject_duplicates(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err)) {\n+\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/* main backend commit */\n \tif (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto done;\n@@ -1153,8 +1260,35 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \tret = the_refs_backend->transaction_commit(transaction,\n \t\t\t\t\t\t &affected_refnames, err);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (the_refs_backend != &refs_be_files) {\n+\t\tret = refs_be_files.transaction_commit(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err);\n+\t\tif (ret) {\n+\t\t\twarning(split_transaction_fail_warning);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/* reflogging for dereferenced symbolic refs */\n+\t\tfor_each_string_list_item(item, &symrefs) {\n+\t\t\tstruct ref_update *update = item->util;\n+\t\t\tif (files_log_ref_write(item->string, update->old_sha1,\n+\t\t\t\t\t\tupdate->new_sha1,\n+\t\t\t\t\t\tupdate->msg, update->flags, err))\n+\t\t\t\twarning(\"failed to log ref update for symref %s\",\n+\t\t\t\t\titem->string);\n+\t\t}\n+\t}\n+\n done:\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&files_affected_refnames, 0);\n+\tif (files_transaction)\n+\t\tref_transaction_free(files_transaction);\n+\tstring_list_clear(&symrefs, 0);\n \treturn ret;\n }\n \n@@ -1210,6 +1344,9 @@ int peel_ref(const char *refname, unsigned char *sha1)\n int create_symref(const char *ref_target, const char *refs_heads_master,\n \t\t const char *logmsg)\n {\n+\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.create_symref(ref_target, refs_heads_master,\n+\t\t\t\t\t\t logmsg);\n \treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n \t\t\t\t\t logmsg);\n }\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273954","messageId":"1449102921-7707-14-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:18Z","receivedAt":"2015-12-03T00:35:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"git init learns a new argument --refs-backend-type. Presently, only\n\"files\" is supported, but later we will add other backends.\n\nWhen this argument is used, the repository's core.refsBackendType\nconfiguration value is set, and the refs backend's initdb function is\nused to set up the ref database.\n\nSigned-off-by: David Turner \n---\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 6 +++++-\n builtin/init-db.c | 10 ++++++++++\n cache.h | 2 ++\n config.c | 20 ++++++++++++++++++++\n environment.c | 1 +\n path.c | 32 ++++++++++++++++++++++++++++++--\n refs.c | 8 ++++++++\n refs.h | 12 ++++++++++++\n setup.c | 10 ++++++++++\n 10 files changed, 99 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt\nindex 648a6cd..72fbd71 100644\n--- a/Documentation/git-init-db.txt\n+++ b/Documentation/git-init-db.txt\n@@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n SYNOPSIS\n --------\n [verse]\n-'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]]\n+'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]] [--refs-backend-type=]\n \n \n DESCRIPTION\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex 8174d27..9ea6753 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git init' [-q | --quiet] [--bare] [--template=]\n \t [--separate-git-dir ]\n \t [--shared[=]] [directory]\n-\n+\t [--refs-backend-type=]\n \n DESCRIPTION\n -----------\n@@ -113,6 +113,10 @@ does not exist, it will be created.\n \n --\n \n+--refs-backend-type=::\n+Type of refs backend. Default is to use the original \"files\" backend,\n+which stores ref data in files in .git/refs and .git/packed-refs.\n+\n TEMPLATE DIRECTORY\n ------------------\n \ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 4771e7e..44db591 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -204,6 +204,14 @@ static int create_default_files(const char *template_path)\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\tif (refs_backend_type) {\n+\t\tstruct refdb_config_data config_data = {NULL};\n+\t\tgit_config_set(\"core.refsBackendType\", refs_backend_type);\n+\t\tconfig_data.refs_backend_type = refs_backend_type;\n+\t\tconfig_data.refs_base = get_git_dir();\n+\t\tset_refs_backend(refs_backend_type, &config_data);\n+\t}\n+\n \tif (refs_init_db(&err, shared_repository))\n \t\tdie(\"failed to set up refs db: %s\", err.buf);\n \n@@ -469,6 +477,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n+\t\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n+\t\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/cache.h b/cache.h\nindex 707455a..d1534db 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -696,6 +696,8 @@ extern enum object_creation_mode object_creation_mode;\n \n extern char *notes_ref_name;\n \n+extern const char *refs_backend_type;\n+\n extern int grafts_replace_parents;\n \n /*\ndiff --git a/config.c b/config.c\nindex 248a21a..210aa08 100644\n--- a/config.c\n+++ b/config.c\n@@ -10,6 +10,7 @@\n #include \"exec_cmd.h\"\n #include \"strbuf.h\"\n #include \"quote.h\"\n+#include \"refs.h\"\n #include \"hashmap.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n@@ -1207,6 +1208,25 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n \t}\n \n \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n+\t\tstruct refdb_config_data refdb_data = {NULL};\n+\t\tchar *repo_config_copy;\n+\n+\t\t/*\n+\t\t * make sure we always read the backend config from the\n+\t\t * core section on startup\n+\t\t */\n+\t\tret += git_config_from_file(refdb_config, repo_config,\n+\t\t\t\t\t &refdb_data);\n+\n+\t\trepo_config_copy = xstrdup(repo_config);\n+\t\trefdb_data.refs_base = xstrdup(dirname(repo_config_copy));\n+\t\tfree(repo_config_copy);\n+\n+\t\tif (refdb_data.refs_backend_type &&\n+\t\t strcmp(refdb_data.refs_backend_type, \"files\")) {\n+\t\t\tdie(\"Unexpected backend %s\", refdb_data.refs_backend_type);\n+\t\t}\n+\n \t\tret += git_config_from_file(fn, repo_config, data);\n \t\tfound += 1;\n \t}\ndiff --git a/environment.c b/environment.c\nindex 2da7fe2..8dbf0ab 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -66,6 +66,7 @@ int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+const char *refs_backend_type;\n \n #ifndef PROTECT_HFS_DEFAULT\n #define PROTECT_HFS_DEFAULT 0\ndiff --git a/path.c b/path.c\nindex 3cd155e..86a8035 100644\n--- a/path.c\n+++ b/path.c\n@@ -2,6 +2,7 @@\n * Utilities for paths and pathnames\n */\n #include \"cache.h\"\n+#include \"refs.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"dir.h\"\n@@ -510,9 +511,36 @@ int validate_headref(const char *path)\n \tunsigned char sha1[20];\n \tint fd;\n \tssize_t len;\n+\tstruct refdb_config_data refdb_data = {NULL, NULL};\n+\n+\tif (lstat(path, &st) < 0) {\n+\t\tint backend_type_set;\n+\t\tstruct strbuf config_path = STRBUF_INIT;\n+\t\tif (path) {\n+\t\t\tchar *pathdup = xstrdup(path);\n+\t\t\tchar *git_dir = dirname(pathdup);\n+\t\t\tstrbuf_addf(&config_path, \"%s/%s\", git_dir, \"config\");\n+\t\t\tfree(pathdup);\n+\t\t} else {\n+\t\t\tstrbuf_addstr(&config_path, \"config\");\n+\t\t}\n \n-\tif (lstat(path, &st) < 0)\n-\t\treturn -1;\n+\t\tif (git_config_from_file(refdb_config, config_path.buf, &refdb_data)) {\n+\t\t\tstrbuf_release(&config_path);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tbackend_type_set = !!refdb_data.refs_backend_type;\n+\t\tfree((void *)refdb_data.refs_backend_type);\n+\t\tfree((void *)refdb_data.refs_base);\n+\t\tstrbuf_release(&config_path);\n+\t\t/*\n+\t\t * Alternate backends are assumed to keep HEAD\n+\t\t * in a valid state, so there's no need to do\n+\t\t * further validation.\n+\t\t */\n+\t\treturn backend_type_set ? 0 : -1;\n+\t}\n \n \t/* Make sure it is a \"refs/..\" symlink */\n \tif (S_ISLNK(st.st_mode)) {\ndiff --git a/refs.c b/refs.c\nindex e48e43a..96e1673 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -24,6 +24,14 @@ struct ref_be *the_refs_backend = &refs_be_files;\n */\n struct ref_be *refs_backends = &refs_be_files;\n \n+const char *refs_backend_type;\n+\n+void register_refs_backend(struct ref_be *be)\n+{\n+\tbe->next = refs_backends;\n+\trefs_backends = be;\n+}\n+\n /*\n * This function is used to switch to an alternate backend.\n */\ndiff --git a/refs.h b/refs.h\nindex c211b9e..c3670e8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -510,6 +510,18 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n+struct refdb_config_data {\n+\tconst char *refs_backend_type;\n+\tconst char *refs_base;\n+};\n+/*\n+ * Read the refdb configuration data out of the config file\n+ */\n+int refdb_config(const char *var, const char *value, void *ptr);\n+\n+struct ref_be;\n int set_refs_backend(const char *name, void *data);\n \n+void register_refs_backend(struct ref_be *be);\n+\n #endif /* REFS_H */\ndiff --git a/setup.c b/setup.c\nindex d343725..de6b8ac 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"refs.h\"\n #include \"string-list.h\"\n \n static int inside_git_dir = -1;\n@@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n \treturn ret;\n }\n \n+int refdb_config(const char *var, const char *value, void *ptr)\n+{\n+ struct refdb_config_data *cdata = ptr;\n+\n+ if (!strcmp(var, \"core.refsbackendtype\"))\n+\t cdata->refs_backend_type = xstrdup((char *)value);\n+ return 0;\n+}\n+\n /*\n * Test if it looks like we're at a git directory.\n * We want to see:\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273956","messageId":"1449102921-7707-15-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 14/16] refs: allow ref backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:19Z","receivedAt":"2015-12-03T00:35:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a new option, --refs-backend-type, to allow the ref backend type to\nbe set on new clones.\n\nSubmodules must use the same ref backend as the parent repository, so\nwe also pass the --refs-backend-type option option when cloning\nsubmodules.\n\nSigned-off-by: David Turner \n---\n Documentation/git-clone.txt | 4 ++++\n builtin/clone.c | 27 +++++++++++++++++++++++++--\n builtin/submodule--helper.c | 5 ++++-\n cache.h | 1 +\n 4 files changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 6bf000d..431575b 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t [-o ] [-b ] [-u ] [--reference ]\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n+\t [--refs-backend-type=]\n \t [--recursive | --recurse-submodules] [--] \n \t []\n \n@@ -221,6 +222,9 @@ objects from the source repository into a pack in the cloned repository.\n \tThe result is Git repository can be separated from working\n \ttree.\n \n+--refs-backend-type=::\n+\tType of refs backend. Default is to use the original files based\n+\tbackend.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex caae43e..a53f341 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -92,11 +92,13 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n \t\t\tN_(\"set config inside the new repository\")),\n+\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n+\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \tOPT_END()\n };\n \n static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n+\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL, NULL\n };\n \n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n@@ -724,8 +726,24 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n+\tif (!err && option_recursive) {\n+\t\tconst char **backend_arg = argv_submodule;\n+\t\tchar *new_backend_arg = NULL;\n+\t\tif (refs_backend_type) {\n+\t\t\twhile (*backend_arg)\n+\t\t\t\t++backend_arg;\n+\n+\t\t\tnew_backend_arg = xmalloc(21 + strlen(refs_backend_type));\n+\t\t\tsprintf(new_backend_arg, \"--refs-backend-type=%s\",\n+\t\t\t\trefs_backend_type);\n+\t\t\t*backend_arg = new_backend_arg;\n+\t\t}\n \t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\t\tif (refs_backend_type) {\n+\t\t\tfree(new_backend_arg);\n+\t\t\t*backend_arg = NULL;\n+\t\t}\n+\t}\n \n \treturn err;\n }\n@@ -744,6 +762,11 @@ static void write_config(struct string_list *config)\n \t\t\t\t\t write_one_config, NULL) < 0)\n \t\t\tdie(\"unable to write parameters to config file\");\n \t}\n+\n+\tif (refs_backend_type &&\n+\t write_one_config(\"core.refsBackendType\",\n+\t\t\t refs_backend_type, NULL) < 0)\n+\t\tdie(\"unable to write backend parameter to config file\");\n }\n \n static void write_refspec_config(const char *src_ref_prefix,\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex f4c3eff..5c9ca4e 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -140,7 +140,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url\n \t\targv_array_pushl(&cp.args, \"--reference\", reference, NULL);\n \tif (gitdir && *gitdir)\n \t\targv_array_pushl(&cp.args, \"--separate-git-dir\", gitdir, NULL);\n-\n+\tif (refs_backend_type && *refs_backend_type) {\n+\t\targv_array_push(&cp.args, \"--refs-backend-type\");\n+\t\targv_array_push(&cp.args, refs_backend_type);\n+\t}\n \targv_array_push(&cp.args, url);\n \targv_array_push(&cp.args, path);\n \ndiff --git a/cache.h b/cache.h\nindex d1534db..8f2ca55 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -695,6 +695,7 @@ enum object_creation_mode {\n extern enum object_creation_mode object_creation_mode;\n \n extern char *notes_ref_name;\n+extern const char *refs_backend_type;\n \n extern const char *refs_backend_type;\n \n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273953","messageId":"1449102921-7707-16-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 15/16] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:20Z","receivedAt":"2015-12-03T00:35:20Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a database backend for refs using LMDB. This backend runs git\nfor-each-ref about 30% faster than the files backend with fully-packed\nrefs on a repo with ~120k refs. It's also about 4x faster than using\nfully-unpacked refs. In addition, and perhaps more importantly, it\navoids case-conflict issues on OS X.\n\nLMDB has a few features that make it suitable for usage in git:\n\n1. It is relatively lightweight; it requires only one header file, and\nthe library code takes under 64k at runtime.\n\n2. It is well-tested: it's been used in OpenLDAP for years.\n\n3. It's very fast. LMDB's benchmarks show that it is among\nthe fastest key-value stores.\n\n4. It has a relatively simple concurrency story; readers don't\nblock writers and writers don't block readers.\n\nRonnie Sahlberg's original version of this patchset used tdb. The\nmajor disadvantage of tdb is that tdb is hard to build on OS X. It's\nalso not in homebrew. So lmdb seemed simpler.\n\nTo test this backend's correctness, I hacked test-lib.sh and\ntest-lib-functions.sh to run all tests under the refs backend. Dozens\nof tests use manual ref/reflog reading/writing, or create submodules\nwithout passing --refs-backend-type to git init. If those tests are\nchanged to use the update-ref machinery or test-refs-lmdb-backend (or,\nin the case of packed-refs, corrupt refs, and dumb fetch tests, are\nskipped), the only remaining failing tests are the git-new-workdir\ntests and the gitweb tests.\n\nSigned-off-by: David Turner \n---\n .gitignore | 1 +\n Documentation/config.txt | 7 +\n Documentation/git-clone.txt | 2 +-\n Documentation/git-init.txt | 3 +-\n Documentation/technical/refs-lmdb-backend.txt | 50 +\n Documentation/technical/repository-version.txt | 5 +\n Makefile | 12 +\n builtin/init-db.c | 10 +-\n config.c | 20 +-\n configure.ac | 33 +\n contrib/workdir/git-new-workdir | 3 +\n refs.h | 1 +\n refs/lmdb-backend.c | 2054 ++++++++++++++++++++++++\n setup.c | 22 +-\n test-refs-lmdb-backend.c | 68 +\n 15 files changed, 2279 insertions(+), 12 deletions(-)\n create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n create mode 100644 refs/lmdb-backend.c\n create mode 100644 test-refs-lmdb-backend.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 1c2f832..87d45a2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -199,6 +199,7 @@\n /test-path-utils\n /test-prio-queue\n /test-read-cache\n+/test-refs-lmdb-backend\n /test-regex\n /test-revision-walking\n /test-run-command\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f617886..5fb25ed 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -496,6 +496,13 @@ core.repositoryFormatVersion::\n \tInternal variable identifying the repository format and layout\n \tversion.\n \n+core.refsBackendType::\n+\tType of refs backend. Default is to use the original files\n+\tbased backend. Set to 'lmdb' to activate the lmdb database\n+\tbackend. If you use the lmdb backend,\n+\tcore.repositoryFormatVersion must be set to 1, and\n+\textensions.refBackend must be set to 'lmdb'.\n+\n core.sharedRepository::\n \tWhen 'group' (or 'true'), the repository is made shareable between\n \tseveral users in a group (making sure all the files and objects are\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 431575b..739c116 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -224,7 +224,7 @@ objects from the source repository into a pack in the cloned repository.\n \n --refs-backend-type=::\n \tType of refs backend. Default is to use the original files based\n-\tbackend.\n+\tbackend. Set to \"lmdb\" to activate the lmdb database backend.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex 9ea6753..bbe253f 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -115,7 +115,8 @@ does not exist, it will be created.\n \n --refs-backend-type=::\n Type of refs backend. Default is to use the original \"files\" backend,\n-which stores ref data in files in .git/refs and .git/packed-refs.\n+which stores ref data in files in .git/refs and .git/packed-refs. Set\n+to \"lmdb\" to activate the lmdb database backend.\n \n TEMPLATE DIRECTORY\n ------------------\ndiff --git a/Documentation/technical/refs-lmdb-backend.txt b/Documentation/technical/refs-lmdb-backend.txt\nnew file mode 100644\nindex 0000000..c497ffc\n--- /dev/null\n+++ b/Documentation/technical/refs-lmdb-backend.txt\n@@ -0,0 +1,50 @@\n+Notes on the LMDB refs backend\n+==============================\n+\n+Design:\n+------\n+\n+Refs and reflogs are stored in a lmdb database in .git/refdb. All\n+keys and values are \\0-terminated.\n+\n+Keys for refs are the name of the ref (e.g. refs/heads/master).\n+Values are the value of the ref, in hex\n+(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n+\n+All per-worktree refs (refs/bisect/* and HEAD) are store using\n+the traditional files-based backend.\n+\n+Reflogs are stored as a series of database entries.\n+\n+For non-empty reflogs, there is one entry per logged ref\n+update. The key format is logs/[refname]\\0[timestamp]. The timestamp\n+is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n+This means that reflog entries are chronologically ordered. Because\n+LMDB is a btree database, we can efficiently iterate over these keys.\n+\n+For an empty reflog, there is a \"header\" entry to show that a reflog\n+exists. The header has the same format as an ordinary reflog, but with\n+a timeztamp of all zeros and an empty value.\n+\n+Reflog values are in the same format as the original files-based\n+reflog.\n+\n+Weaknesses:\n+-----------\n+\n+The reflog format is somewhat inefficient: a binary format could store\n+reflog date/time information in somewhat less space.\n+\n+The rsync and file:// transports don't work yet, because they\n+don't use the refs API.\n+\n+git new-workdir is incompatible with the lmdb backend. Fortunately,\n+git new-workdir is deprecated, and worktrees work fine.\n+\n+LMDB locks the entire database for write. Any other writer waits\n+until the first writer is done before beginning. Readers do not wait\n+for writers, and writers do not wait for readers. The underlying\n+scheme is approximately MVCC; each reader's queries see the state of\n+the database as-of the time that the reader acquired its read lock.\n+This is not too far off from the files backend, which loads all refs\n+into memory when one is requested.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 00ad379..04c085d 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -86,3 +86,8 @@ for testing format-1 compatibility.\n When the config key `extensions.preciousObjects` is set to `true`,\n objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n `git repack -d`).\n+\n+`refBackend`\n+~~~~~~~~~~~~\n+This extension allows the user of alternate ref backends. The only\n+defined value is `lmdb`.\ndiff --git a/Makefile b/Makefile\nindex 5bd68e0..77b96d9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1037,6 +1037,17 @@ ifdef USE_LIBPCRE\n \tEXTLIBS += -lpcre\n endif\n \n+ifdef USE_LIBLMDB\n+\tBASIC_CFLAGS += -DUSE_LIBLMDB\n+\tifdef LIBLMDBDIR\n+\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n+\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n+\tendif\n+\tEXTLIBS += -llmdb\n+\tLIB_OBJS += refs/lmdb-backend.o\n+\tTEST_PROGRAMS_NEED_X += test-refs-lmdb-backend\n+endif\n+\n ifdef HAVE_ALLOCA_H\n \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n endif\n@@ -2124,6 +2135,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n+\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 44db591..1eb2feb 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -179,6 +179,7 @@ static int create_default_files(const char *template_path)\n \tint reinit;\n \tint filemode;\n \tstruct strbuf err = STRBUF_INIT;\n+\tint repo_version = 0;\n \n \t/* Just look for `init.templatedir` */\n \tgit_config(git_init_db_config, NULL);\n@@ -209,7 +210,14 @@ static int create_default_files(const char *template_path)\n \t\tgit_config_set(\"core.refsBackendType\", refs_backend_type);\n \t\tconfig_data.refs_backend_type = refs_backend_type;\n \t\tconfig_data.refs_base = get_git_dir();\n+#ifdef USE_LIBLMDB\n+\t\tregister_refs_backend(&refs_be_lmdb);\n+#endif\n \t\tset_refs_backend(refs_backend_type, &config_data);\n+\t\tif (!strcmp(refs_backend_type, \"lmdb\")) {\n+\t\t\tgit_config_set(\"extensions.refbackend\", \"lmdb\");\n+\t\t\trepo_version = 1;\n+\t\t}\n \t}\n \n \tif (refs_init_db(&err, shared_repository))\n@@ -229,7 +237,7 @@ static int create_default_files(const char *template_path)\n \n \t/* This forces creation of new config file */\n \txsnprintf(repo_version_string, sizeof(repo_version_string),\n-\t\t \"%d\", GIT_REPO_VERSION);\n+\t\t \"%d\", repo_version);\n \tgit_config_set(\"core.repositoryformatversion\", repo_version_string);\n \n \t/* Check filemode trustability */\ndiff --git a/config.c b/config.c\nindex 210aa08..779bb73 100644\n--- a/config.c\n+++ b/config.c\n@@ -1222,9 +1222,23 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n \t\trefdb_data.refs_base = xstrdup(dirname(repo_config_copy));\n \t\tfree(repo_config_copy);\n \n-\t\tif (refdb_data.refs_backend_type &&\n-\t\t strcmp(refdb_data.refs_backend_type, \"files\")) {\n-\t\t\tdie(\"Unexpected backend %s\", refdb_data.refs_backend_type);\n+\t\tif (!refdb_data.refs_backend_type)\n+\t\t\trefdb_data.refs_backend_type = \"\";\n+\n+\t\tif ((!*refdb_data.refs_backend_type) ||\n+\t\t (!strcmp(refdb_data.refs_backend_type, \"files\"))) {\n+\t\t\t/* default backend, nothing to do */\n+\t\t} else if (!strcmp(refdb_data.refs_backend_type, \"lmdb\")) {\n+\n+#ifdef USE_LIBLMDB\n+\t\t\trefs_backend_type = refdb_data.refs_backend_type;\n+\t\t\tregister_refs_backend(&refs_be_lmdb);\n+\t\t\tset_refs_backend(refs_backend_type, &refdb_data);\n+#else\n+\t\t\tdie(\"Git was not built with USE_LIBLMDB, so the db refs backend is not available\");\n+#endif\n+\t\t} else {\n+\t\t\tdie(\"Unknown ref backend type '%s'\", refdb_data.refs_backend_type);\n \t\t}\n \n \t\tret += git_config_from_file(fn, repo_config, data);\ndiff --git a/configure.ac b/configure.ac\nindex 89e2590..3853bec 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -271,6 +271,24 @@ AS_HELP_STRING([], [ARG can be also prefix for libpcre library and hea\n dnl it yet.\n \tGIT_CONF_SUBST([LIBPCREDIR])\n fi)\n+\n+USE_LIBLMDB=YesPlease\n+AC_ARG_WITH(liblmdb,\n+AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])\n+AS_HELP_STRING([], [ARG can be also prefix for liblmdb library and headers]),\n+ if test \"$withval\" = \"no\"; then\n+\tUSE_LIBLMDB=\n+ elif test \"$withval\" = \"yes\"; then\n+\tUSE_LIBLMDB=YesPlease\n+ else\n+\tUSE_LIBLMDB=YesPlease\n+\tLIBLMDBDIR=$withval\n+\tAC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])\n+ dnl USE_LIBLMDB can still be modified below, so don't substitute\n+ dnl it yet.\n+\tGIT_CONF_SUBST([LIBLMDBDIR])\n+ fi)\n+\n #\n # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n AC_FUNC_ALLOCA\n@@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])\n \n fi\n \n+if test -n \"$USE_LIBLMDB\"; then\n+\n+GIT_STASH_FLAGS($LIBLMDBDIR)\n+\n+AC_CHECK_LIB([lmdb], [mdb_env_open],\n+[USE_LIBLMDB=YesPlease],\n+[USE_LIBLMDB=])\n+\n+GIT_UNSTASH_FLAGS($LIBLMDBDIR)\n+\n+GIT_CONF_SUBST([USE_LIBLMDB])\n+\n+fi\n+\n+\n #\n # Define NO_CURL if you do not have libcurl installed. git-http-pull and\n # git-http-push are not built, and you cannot use http:// and https://\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a..66b7ecf 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -28,6 +28,9 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n git rev-parse --git-dir 2>/dev/null) ||\n die \"Not a git repository: \\\"$orig_git\\\"\"\n \n+\n+test \"$(git config core.refsbackendtype)\" = \"lmdb\" && die \"git-new-workdir is incompatible with the refs lmdb backend\"\n+\n case \"$git_dir\" in\n .git)\n \tgit_dir=\"$orig_git/.git\"\ndiff --git a/refs.h b/refs.h\nindex c3670e8..0cbfda9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -520,6 +520,7 @@ struct refdb_config_data {\n int refdb_config(const char *var, const char *value, void *ptr);\n \n struct ref_be;\n+extern struct ref_be refs_be_lmdb;\n int set_refs_backend(const char *name, void *data);\n \n void register_refs_backend(struct ref_be *be);\ndiff --git a/refs/lmdb-backend.c b/refs/lmdb-backend.c\nnew file mode 100644\nindex 0000000..654048b\n--- /dev/null\n+++ b/refs/lmdb-backend.c\n@@ -0,0 +1,2054 @@\n+/*\n+ * This file implements a lmdb backend for refs.\n+ *\n+ * The design of this backend relies on lmdb's write lock -- that is, any\n+ * write transaction blocks all other writers. Thus, as soon as a ref\n+ * transaction is opened, we know that any values we read won't\n+ * change out from under us, and we have a fully-consistent view of the\n+ * database.\n+ *\n+ * We store the content of refs including the trailing \\0 so that\n+ * standard C string functions can handle them. Just like struct\n+ * strbuf.\n+ */\n+#include \"../cache.h\"\n+#include \n+#include \"../object.h\"\n+#include \"../refs.h\"\n+#include \"refs-internal.h\"\n+#include \"../tag.h\"\n+#include \"../lockfile.h\"\n+\n+static struct trace_key db_trace = TRACE_KEY_INIT(LMDB);\n+\n+static MDB_env *env;\n+\n+static char *db_path;\n+\n+extern struct ref_be refs_be_files;\n+\n+struct lmdb_transaction {\n+\tMDB_txn *txn;\n+\tMDB_dbi dbi;\n+\tMDB_cursor *cursor;\n+\tconst char *submodule;\n+\tint flags;\n+};\n+\n+struct lmdb_transaction transaction;\n+\n+static char *get_refdb_path(const char *base)\n+{\n+\tstruct strbuf path_buf = STRBUF_INIT;\n+\tstrbuf_addf(&path_buf, \"%s/refdb\", base);\n+\treturn strbuf_detach(&path_buf, NULL);\n+}\n+\n+static int in_write_transaction(void)\n+{\n+\treturn transaction.txn && !(transaction.flags & MDB_RDONLY);\n+}\n+\n+static void init_env(MDB_env **env, const char *path)\n+{\n+\tint ret;\n+\tif (*env)\n+\t\treturn;\n+\n+\tret = mdb_env_create(env);\n+\tif (ret)\n+\t\tdie(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_set_maxreaders(*env, 1000);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_env_set_maxreaders failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_set_mapsize(*env, (1<<30));\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_set_mapsize failed: %s\", mdb_strerror(ret));\n+\tret = mdb_env_open(*env, path, 0 , 0664);\n+\tif (ret)\n+\t\tdie(\"BUG: mdb_env_open (%s) failed: %s\", path,\n+\t\t mdb_strerror(ret));\n+}\n+\n+static int lmdb_init_db(struct strbuf *err, int shared)\n+{\n+\t/*\n+\t * To create a db, all we need to do is make a directory for\n+\t * it to live in; lmdb will do the rest.\n+\t */\n+\n+\tassert(db_path);\n+\tif (mkdir(db_path, 0775) && errno != EEXIST) {\n+\t\tstrbuf_addf(err, \"%s\", strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void lmdb_init_backend(void *cbdata)\n+{\n+\tstruct refdb_config_data *data = (struct refdb_config_data *)cbdata;\n+\n+\tif (db_path)\n+\t\treturn;\n+\n+\tdb_path = xstrdup(real_path(get_refdb_path(data->refs_base)));\n+\n+\trefs_be_files.init_backend(NULL);\n+\ttrace_printf_key(&db_trace, \"Init backend\\n\");\n+}\n+\n+static void mdb_cursor_open_or_die(struct lmdb_transaction *transaction,\n+\t\t\t\t MDB_cursor **cursor)\n+{\n+\tint ret = mdb_cursor_open(transaction->txn, transaction->dbi, cursor);\n+\tif (ret)\n+\t\tdie(\"mdb_cursor_open failed: %s\", mdb_strerror(ret));\n+}\n+\n+static void submodule_path(struct strbuf *sb, const char *submodule,\n+\t\t\t const char *refname)\n+{\n+\tif (submodule)\n+\t\tstrbuf_git_path_submodule(sb, submodule, \"%s\", refname);\n+\telse\n+\t\tstrbuf_git_path(sb, \"%s\", refname);\n+}\n+\n+static int read_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t struct MDB_val *val, int *needs_free)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct stat st;\n+\tint ret = -1;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+#ifndef NO_SYMLINK_HEAD\n+\tif (lstat(path.buf, &st)) {\n+\t\tif (errno == ENOENT)\n+\t\t\tret = MDB_NOTFOUND;\n+\t\tgoto done;\n+\t}\n+\tif (S_ISLNK(st.st_mode)) {\n+\t\tstrbuf_readlink(&sb, path.buf, 0);\n+\t\tif (starts_with(sb.buf, \"refs/\") &&\n+\t\t !check_refname_format(sb.buf, 0)) {\n+\t\t\tval->mv_data = xmalloc(sb.len + 6);\n+\t\t\tval->mv_size = sprintf(val->mv_data, \"ref: %s\",\n+\t\t\t\t\t sb.buf) + 1;\n+\t\t\tret = 0;\n+\t\t} else {\n+\t\t\tret = MDB_NOTFOUND;\n+\t\t}\n+\t\tstrbuf_release(&sb);\n+\t\tgoto done;\n+\t}\n+#endif\n+\n+\tif (strbuf_read_file(&sb, path.buf, 200) < 0) {\n+\t\tstrbuf_release(&sb);\n+\t\tif (errno == ENOENT)\n+\t\t\tret = MDB_NOTFOUND;\n+\t\tgoto done;\n+\t}\n+\tstrbuf_rtrim(&sb);\n+\n+\tval->mv_data = strbuf_detach(&sb, &val->mv_size);\n+\tval->mv_size++;\n+\n+\tret = 0;\n+done:\n+\tstrbuf_release(&path);\n+\t*needs_free = !ret;\n+\treturn ret;\n+}\n+\n+static void write_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t MDB_val *val)\n+{\n+\tstatic struct lock_file lock;\n+\tint fd;\n+\tint len = val->mv_size - 1;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\tsafe_create_leading_directories(path.buf);\n+\n+\tfd = hold_lock_file_for_update(&lock, path.buf, LOCK_DIE_ON_ERROR);\n+\tstrbuf_release(&path);\n+\n+\tif (write_in_full(fd, val->mv_data, len) != len ||\n+\t write_in_full(fd, \"\\n\", 1) != 1)\n+\t\tdie_errno(\"failed to write new HEAD\");\n+\n+\tif (commit_lock_file(&lock))\n+\t\tdie_errno(\"failed to write new HEAD\");\n+}\n+\n+static int del_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\tMDB_val *val)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint result;\n+\n+\t/*\n+\t * Returning deleted ref data is not yet implemented, but no\n+\t * callers need it.\n+\t */\n+\tassert(val == NULL);\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+\tresult = unlink(path.buf);\n+\tstrbuf_release(&path);\n+\tif (result && errno != ENOENT)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Read a ref. If the ref is a per-worktree ref, read it from disk.\n+ * Otherwise, read it from LMDB. LMDB manages its own memory, so the\n+ * data returned in *val will ordinarily not need to be freed. But\n+ * when a per-worktree ref is (successfully) read, non-LMDB memory is\n+ * allocated. In this case, *needs_free is set so that the caller can\n+ * free the memory when it is done with it.\n+ */\n+static int mdb_get_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val, int *needs_free)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\treturn read_per_worktree_ref(transaction->submodule,\n+\t\t\t\t\t key->mv_data, val, needs_free);\n+\n+\t*needs_free = 0;\n+\tret = mdb_get(transaction->txn, transaction->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static int mdb_del_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\tdie(\"BUG: this backend should only try to delete normal refs\");\n+\n+\tret = mdb_del(transaction->txn, transaction->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static void mdb_put_or_die(struct lmdb_transaction *transaction, MDB_val *key,\n+\t\t\t MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\tdie(\"BUG: this backend should only try to write normal refs\");\n+\n+\tret = mdb_put(transaction->txn, transaction->dbi, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret == MDB_BAD_VALSIZE)\n+\t\t\tdie(\"Ref name %s too long (max size is %d)\",\n+\t\t\t (const char *)key->mv_data,\n+\t\t\t mdb_env_get_maxkeysize(env));\n+\t\telse\n+\t\t\tdie(\"mdb_put failed: (%s -> %s) %s\",\n+\t\t\t (const char *)key->mv_data,\n+\t\t\t (const char *)val->mv_data, mdb_strerror(ret));\n+\t}\n+}\n+\n+static int mdb_cursor_get_or_die(MDB_cursor *cursor, MDB_val *key, MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tret = mdb_cursor_get(cursor, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_cursor_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\tassert(((char *)val->mv_data)[val->mv_size - 1] == 0);\n+\treturn 0;\n+}\n+\n+static int mdb_cursor_del_or_die(MDB_cursor *cursor, int flags)\n+{\n+\tint ret = mdb_cursor_del(cursor, flags);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_cursor_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * Begin a transaction. Because only one transaction per thread is\n+ * permitted, we use a global transaction object. If a read-write\n+ * transaction is presently already in-progress, and a read-only\n+ * transaction is requested, the read-write transaction will be\n+ * returned instead. If a read-write transaction is requested and a\n+ * read-only transaction is open, the read-only transaction will be\n+ * closed.\n+ *\n+ * It is a bug to request a read-write transaction during another\n+ * read-write transaction.\n+ *\n+ * As a result, it is unsafe to retain read-only transactions past the\n+ * point where a read-write transaction might be needed. For\n+ * instance, any call that has callbacks outside this module must\n+ * conclude all of its reads from the database before calling those\n+ * callbacks, or must reacquire the transaction after its callbacks\n+ * are completed.\n+ */\n+int lmdb_transaction_begin_flags(struct strbuf *err, unsigned int flags)\n+{\n+\tint ret;\n+\tMDB_txn *txn;\n+\tstatic size_t last_txnid = 0;\n+\tint force_restart = 0;\n+\tMDB_envinfo stat;\n+\n+\tinit_env(&env, db_path);\n+\n+\t/*\n+\t * Since each transaction sees a consistent view of the db,\n+\t * downstream processes that write the db won't be seen in\n+\t * this transaction. We can check if the last transaction id\n+\t * has changed since this read transaction was started, and if\n+\t * so, we want to reopen the transaction.\n+\t */\n+\n+\tmdb_env_info(env, &stat);\n+\tif (stat.me_last_txnid != last_txnid) {\n+\t\tforce_restart = 1;\n+\t\tlast_txnid = stat.me_last_txnid;\n+\t}\n+\n+\tif (!transaction.txn) {\n+\t\tret = mdb_txn_begin(env, NULL, flags, &txn);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t\treturn 0;\n+\t}\n+\n+\tif (transaction.flags == flags && !(flags & MDB_RDONLY))\n+\t\tdie(\"BUG: rw transaction started during another rw txn\");\n+\n+\tif (force_restart || (transaction.flags != flags && transaction.flags & MDB_RDONLY)) {\n+\t\t/*\n+\t\t * RO -> RW, or forced restart due to possible changes\n+\t\t * from downstream processes.\n+\t\t */\n+\t\tmdb_txn_abort(transaction.txn);\n+\t\tret = mdb_txn_begin(env, NULL, flags, &txn);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"restarting txn: mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);\n+\t\tif (ret) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t}\n+\t/* RW -> RO just keeps the RW txn */\n+\treturn 0;\n+}\n+\n+static struct lmdb_transaction *lmdb_transaction_begin_flags_or_die(int flags)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tif (lmdb_transaction_begin_flags(&err, flags))\n+\t\tdie(\"%s\", err.buf);\n+\treturn &transaction;\n+}\n+\n+#define MAXDEPTH 5\n+\n+static const char *parse_ref_data(struct lmdb_transaction *transaction,\n+\t\t\t\t const char *refname, const char *ref_data,\n+\t\t\t\t unsigned char *sha1, int resolve_flags,\n+\t\t\t\t int *flags, int bad_name)\n+{\n+\tint depth = MAXDEPTH;\n+\tconst char *buf;\n+\tstatic struct strbuf refname_buffer = STRBUF_INIT;\n+\tstatic struct strbuf refdata_buffer = STRBUF_INIT;\n+\tMDB_val key, val;\n+\tint needs_free = 0;\n+\n+\tfor (;;) {\n+\t\tif (--depth < 0)\n+\t\t\treturn NULL;\n+\n+\t\tif (!starts_with(ref_data, \"ref:\")) {\n+\t\t\tif (get_sha1_hex(ref_data, sha1) ||\n+\t\t\t (ref_data[40] != '\\0' && !isspace(ref_data[40]))) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tif (bad_name) {\n+\t\t\t\thashclr(sha1);\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t} else if (is_null_sha1(sha1)) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (flags)\n+\t\t\t*flags |= REF_ISSYMREF;\n+\t\tbuf = ref_data + 4;\n+\t\twhile (isspace(*buf))\n+\t\t\tbuf++;\n+\t\tstrbuf_reset(&refname_buffer);\n+\t\tstrbuf_addstr(&refname_buffer, buf);\n+\t\trefname = refname_buffer.buf;\n+\t\tif (resolve_flags & RESOLVE_REF_NO_RECURSE) {\n+\t\t\thashclr(sha1);\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\n+\t\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t\t !refname_is_safe(buf)) {\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\tbad_name = 1;\n+\t\t}\n+\n+\t\tkey.mv_data = (char *)refname;\n+\t\tkey.mv_size = strlen(refname) + 1;\n+\t\tif (mdb_get_or_die(transaction, &key, &val, &needs_free)) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (bad_name) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\t\treturn NULL;\n+\n+\t\t\treturn refname;\n+\t\t}\n+\t\tstrbuf_reset(&refdata_buffer);\n+\t\tstrbuf_add(&refdata_buffer, val.mv_data, val.mv_size);\n+\t\tif (needs_free)\n+\t\t\tfree(val.mv_data);\n+\t\tref_data = refdata_buffer.buf;\n+\t}\n+\treturn refname;\n+}\n+\n+static int verify_refname_available_txn(struct lmdb_transaction *transaction,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tstruct string_list *extras,\n+\t\t\t\t\tstruct string_list *skip,\n+\t\t\t\t\tstruct strbuf *err)\n+{\n+\tMDB_cursor *cursor;\n+\tMDB_val key;\n+\tMDB_val val;\n+\tint mdb_ret;\n+\tsize_t refname_len;\n+\tchar *search_key;\n+\tconst char *extra_refname;\n+\tint ret = 1;\n+\tsize_t i;\n+\n+\tmdb_cursor_open_or_die(transaction, &cursor);\n+\n+\trefname_len = strlen(refname) + 2;\n+\tkey.mv_size = refname_len;\n+\tsearch_key = xmalloc(refname_len);\n+\tmemcpy(search_key, refname, refname_len - 2);\n+\tsearch_key[refname_len - 2] = '/';\n+\tsearch_key[refname_len - 1] = 0;\n+\tkey.mv_data = search_key;\n+\n+\t/* Check for subdirs of refname: we start at refname/ */\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (starts_with(key.mv_data, refname) &&\n+\t\t ((char*)key.mv_data)[refname_len - 2] == '/') {\n+\t\t\tif (skip && string_list_has_string(skip, key.mv_data))\n+\t\t\t\tgoto next;\n+\n+\t\t\tstrbuf_addf(err, \"'%s' exists; cannot create '%s'\", (char *)key.mv_data, refname);\n+\t\t\tgoto done;\n+\t\t}\n+\t\tbreak;\n+\tnext:\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\t/* Check for parent dirs of refname. */\n+\tfor (i = 0; i < refname_len - 2; i++) {\n+\t\tif (search_key[i] == '/') {\n+\t\t\tsearch_key[i] = 0;\n+\t\t\tif (skip && string_list_has_string(skip, search_key)) {\n+\t\t\t\tsearch_key[i] = '/';\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, search_key)) {\n+\t\t\t\tstrbuf_addf(err, \"cannot process '%s' and '%s' at the same time\",\n+\t\t\t\t\t refname, search_key);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tkey.mv_data = search_key;\n+\t\t\tkey.mv_size = i + 1;\n+\t\t\tif (!mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET)) {\n+\t\t\t\tstrbuf_addf(err, \"'%s' exists; cannot create '%s'\", (char *)key.mv_data, refname);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tsearch_key[i] = '/';\n+\t\t}\n+\t}\n+\n+\textra_refname = find_descendant_ref(refname, extras, skip);\n+\tif (extra_refname) {\n+\t\tstrbuf_addf(err,\n+\t\t\t \"cannot process '%s' and '%s' at the same time\",\n+\t\t\t refname, extra_refname);\n+\t\tret = 1;\n+\t} else {\n+\t\tret = 0;\n+\t}\n+done:\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\treturn ret;\n+}\n+\n+static const char *resolve_ref_unsafe_txn(struct lmdb_transaction *transaction,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n+{\n+\tint bad_name = 0;\n+\tchar *ref_data;\n+\tstruct MDB_val key, val;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint needs_free = 0;\n+\tconst char *ret;\n+\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\n+\tif (flags)\n+\t\t*flags = 0;\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (flags)\n+\t\t\t*flags |= REF_BAD_NAME;\n+\n+\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t !refname_is_safe(refname)) {\n+\t\t\terrno = EINVAL;\n+\t\t\treturn NULL;\n+\t\t}\n+\t\t/*\n+\t\t * dwim_ref() uses REF_ISBROKEN to distinguish between\n+\t\t * missing refs and refs that were present but invalid,\n+\t\t * to complain about the latter to stderr.\n+\t\t *\n+\t\t * We don't know whether the ref exists, so don't set\n+\t\t * REF_ISBROKEN yet.\n+\t\t */\n+\t\tbad_name = 1;\n+\t}\n+\n+\tkey.mv_data = (void *)refname;\n+\tkey.mv_size = strlen(refname) + 1;\n+\tif (mdb_get_or_die(transaction, &key, &val, &needs_free)) {\n+\t\tif (bad_name) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\treturn NULL;\n+\n+\t\tif (verify_refname_available_txn(transaction, refname, NULL, NULL, &err)) {\n+\t\t\terror(\"%s\", err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\thashclr(sha1);\n+\t\treturn refname;\n+\t}\n+\n+\tref_data = val.mv_data;\n+\tassert(ref_data[val.mv_size - 1] == 0);\n+\n+\tret = parse_ref_data(transaction, refname, ref_data, sha1,\n+\t\t\t resolve_flags, flags, bad_name);\n+\tif (needs_free)\n+\t\tfree(ref_data);\n+\treturn ret;\n+}\n+\n+static const char *lmdb_resolve_ref_unsafe(const char *refname, int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn resolve_ref_unsafe_txn(&transaction, refname,\n+\t\t\t\t resolve_flags, sha1, flags);\n+}\n+\n+static void write_u64(char *buf, uint64_t number)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < 8; i++)\n+\t\tbuf[i] = (number >> (i * 8)) & 0xff;\n+}\n+\n+static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tchar *email_end, *message;\n+\tunsigned long timestamp;\n+\tint tz;\n+\n+\t/* old (raw) new (raw) name SP time TAB msg LF */\n+\tif (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n+\t !(email_end = strchr(sb->buf + 40, '>')) ||\n+\t email_end[1] != ' ' ||\n+\t !(timestamp = strtoul(email_end + 2, &message, 10)) ||\n+\t !message || message[0] != ' ' ||\n+\t (message[1] != '+' && message[1] != '-') ||\n+\t !isdigit(message[2]) || !isdigit(message[3]) ||\n+\t !isdigit(message[4]) || !isdigit(message[5]))\n+\t\treturn 0; /* corrupt? */\n+\n+\thashcpy(osha1, (const unsigned char *)sb->buf);\n+\thashcpy(nsha1, (const unsigned char *)sb->buf + 20);\n+\n+\temail_end[1] = '\\0';\n+\ttz = strtol(message + 1, NULL, 10);\n+\tif (message[6] != '\\t')\n+\t\tmessage += 6;\n+\telse\n+\t\tmessage += 7;\n+\treturn fn(osha1, nsha1, sb->buf + 40, timestamp, tz, message, cb_data);\n+}\n+\n+static void format_reflog_entry(struct strbuf *buf,\n+\t\t\t\tconst unsigned char *old_sha1,\n+\t\t\t\tconst unsigned char *new_sha1,\n+\t\t\t\tconst char *committer, const char *msg)\n+{\n+\tint len;\n+\tint msglen;\n+\n+\tassert(buf->len == 0);\n+\tstrbuf_add(buf, old_sha1, 20);\n+\tstrbuf_add(buf, new_sha1, 20);\n+\tstrbuf_addstr(buf, committer);\n+\tstrbuf_addch(buf, '\\n');\n+\n+\tlen = buf->len;\n+\tmsglen = msg ? strlen(msg) : 0;\n+\tif (msglen) {\n+\t\tint copied;\n+\t\tstrbuf_grow(buf, msglen + 1);\n+\t\tcopied = copy_reflog_msg(buf->buf + 40 + strlen(committer), msg) - 1;\n+\t\tbuf->len = len + copied;\n+\t\tbuf->buf[buf->len] = 0;\n+\t}\n+}\n+\n+static int log_ref_write(const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const char *msg,\n+\t\t\t int flags,\n+\t\t\t struct strbuf *err)\n+{\n+\tMDB_val key, val;\n+\tuint64_t now = getnanotime();\n+\tint result;\n+\tchar *log_key;\n+\tint refname_len;\n+\tMDB_cursor *cursor;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *timestamp;\n+\n+\tif (log_all_ref_updates < 0)\n+\t\tlog_all_ref_updates = !is_bare_repository();\n+\n+\t/* it is assumed that we are in a ref transaction here */\n+\tassert(transaction.txn);\n+\n+\tresult = safe_create_reflog(refname, flags & REF_FORCE_CREATE_REFLOG, err);\n+\tif (result)\n+\t\treturn result;\n+\n+\t/* \"logs/\" + refname + \\0 + 8-byte timestamp for sorting and expiry. */\n+\trefname_len = strlen(refname);\n+\tkey.mv_size = refname_len + 14;\n+\tlog_key = xcalloc(1, key.mv_size);\n+\tsprintf(log_key, \"logs/%s\", refname);\n+\tkey.mv_data = log_key;\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\t/* if no reflog exists, we're done */\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||\n+\t strcmp(key.mv_data, log_key))\n+\t\tgoto done;\n+\n+\t/* Is this a header? We only need the header for empty reflogs */\n+\ttimestamp = (const char *)key.mv_data + refname_len + 6;\n+\tif (ntohll(*(uint64_t *)timestamp) == 0)\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\n+\tkey.mv_data = log_key;\n+\n+\twrite_u64((char *)key.mv_data + refname_len + 6, htonll(now));\n+\n+\tformat_reflog_entry(&buf, old_sha1, new_sha1,\n+\t\t\t git_committer_info(0), msg);\n+\tassert(buf.len >= 42);\n+\tval.mv_data = buf.buf;\n+\tval.mv_size = buf.len + 1;\n+\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\tstrbuf_release(&buf);\n+\n+done:\n+\tfree(log_key);\n+\tmdb_cursor_close(cursor);\n+\treturn 0;\n+}\n+\n+static int lmdb_verify_refname_available(const char *refname,\n+\t\t\t\t\t struct string_list *extras,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn verify_refname_available_txn(&transaction, refname, extras, skip, err);\n+}\n+\n+/*\n+ * Attempt to resolve `refname` to `old_sha1` (if old_sha1 is\n+ * non-null). The return value is a pointer to a newly-allocated\n+ * string containing the next ref name that this resolves to. So if\n+ * HEAD is a symbolic ref to refs/heads/example, which is itself a\n+ * symbolic ref to refs/heads/foo, return refs/heads/example,\n+ * and fill in resolved_sha1 with the sha of refs/heads/foo.\n+ */\n+static char *check_ref(MDB_txn *txn, const char *refname,\n+\t\t const unsigned char *old_sha1,\n+\t\t unsigned char *resolved_sha1, int flags,\n+\t\t int *type_p)\n+{\n+\tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n+\tint resolve_flags = 0;\n+\tint type;\n+\tchar *resolved_refname;\n+\n+\tif (mustexist)\n+\t\tresolve_flags |= RESOLVE_REF_READING;\n+\tif (flags & REF_DELETING) {\n+\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n+\t\tif (flags & REF_NODEREF)\n+\t\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n+\t}\n+\n+\t/*\n+\t * The first time we resolve the refname, we're just trying to\n+\t * see if there is any ref at all by this name, even if it is\n+\t * a broken symref.\n+\t */\n+\trefname = resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t resolved_sha1, &type);\n+\tif (type_p)\n+\t *type_p = type;\n+\n+\tif (!refname)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * Need to copy refname here because the resolve_ref_unsafe\n+\t * returns a pointer to a static buffer that could get mangled\n+\t * by the second call.\n+\t */\n+\tresolved_refname = xstrdup(refname);\n+\n+\tif (old_sha1) {\n+\t\tif (flags & REF_NODEREF) {\n+\t\t\tresolve_flags &= ~RESOLVE_REF_NO_RECURSE;\n+\n+\t\t\tresolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t resolved_sha1, &type);\n+\t\t}\n+\t\tif (hashcmp(old_sha1, resolved_sha1)) {\n+\t\t\terror(\"ref %s is at %s but expected %s\", refname,\n+\t\t\t sha1_to_hex(resolved_sha1), sha1_to_hex(old_sha1));\n+\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\treturn resolved_refname;\n+}\n+\n+static int mdb_transaction_commit(struct lmdb_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint result;\n+\n+\tresult = mdb_txn_commit(transaction->txn);\n+\tif (result && err)\n+\t\tstrbuf_addstr(err, mdb_strerror(result));\n+\n+\ttransaction->txn = NULL;\n+\treturn result;\n+}\n+\n+static int lmdb_delete_reflog(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tint mdb_ret;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint in_transaction;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.delete_reflog(refname);\n+\n+\tin_transaction = in_write_transaction();\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) || ((char*)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\ttransaction.cursor = NULL;\n+\n+\tif (!in_transaction && mdb_transaction_commit(&transaction, &err)) {\n+\t\twarning(\"%s\", err.buf);\n+\t\tret = 01;\n+\t}\n+\tstrbuf_release(&err);\n+\treturn ret;\n+}\n+\n+#define REF_NO_REFLOG 0x8000\n+\n+static int lmdb_transaction_update(const char *refname,\n+\t\t\t\t const unsigned char *new_sha1,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tconst char *orig_refname = refname;\n+\tMDB_val key, val;\n+\tunsigned char resolved_sha1[20];\n+\tint type;\n+\tint ret = -1;\n+\n+\tif ((flags & REF_HAVE_NEW) && is_null_sha1(new_sha1))\n+\t\tflags |= REF_DELETING;\n+\n+\tif (new_sha1 && !is_null_sha1(new_sha1) &&\n+\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(err, \"refusing to update ref with bad name %s\",\n+\t\t\t refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\trefname = check_ref(transaction.txn, orig_refname, old_sha1,\n+\t\t\t resolved_sha1, flags, &type);\n+\tif (refname == NULL) {\n+\t\tstrbuf_addf(err, \"cannot lock the ref '%s'\", orig_refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\tif (!(flags & REF_DELETING) && is_null_sha1(resolved_sha1) &&\n+\t verify_refname_available_txn(&transaction, refname, NULL, NULL, err))\n+\t\treturn TRANSACTION_NAME_CONFLICT;\n+\n+\tif (flags & REF_NODEREF) {\n+\t\tfree((void *)refname);\n+\t\trefname = orig_refname;\n+\t}\n+\n+\tkey.mv_size = strlen(refname) + 1;\n+\tkey.mv_data = (void *)refname;\n+\n+\tif ((flags & REF_HAVE_NEW) && !is_null_sha1(new_sha1)) {\n+\t\tint overwriting_symref = ((type & REF_ISSYMREF) &&\n+\t\t\t\t\t (flags & REF_NODEREF));\n+\n+\t\tstruct object *o = parse_object(new_sha1);\n+\t\tif (!o) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Trying to write ref %s with nonexistent object %s\",\n+\t\t\t\t refname, sha1_to_hex(new_sha1));\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (o->type != OBJ_COMMIT && is_branch(refname)) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Trying to write non-commit object %s to branch %s\",\n+\t\t\t\t sha1_to_hex(new_sha1), refname);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (!overwriting_symref\n+\t\t && !hashcmp(resolved_sha1, new_sha1)) {\n+\t\t\t/*\n+\t\t\t * The reference already has the desired\n+\t\t\t * value, so we don't need to write it.\n+\t\t\t */\n+\t\t\tflags |= REF_NO_REFLOG;\n+\t\t} else {\n+\t\t\tval.mv_size = 41;\n+\t\t\tif (new_sha1)\n+\t\t\t\tval.mv_data = sha1_to_hex(new_sha1);\n+\t\t\telse\n+\t\t\t\tval.mv_data = sha1_to_hex(null_sha1);\n+\t\t\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\t\t}\n+\t}\n+\n+\tif (flags & REF_DELETING) {\n+\t\tif (mdb_del_or_die(&transaction, &key, NULL)) {\n+\t\t\tif (old_sha1 && !is_null_sha1(old_sha1)) {\n+\t\t\t\tstrbuf_addf(err, \"No such ref %s\", refname);\n+\t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t\tlmdb_delete_reflog(orig_refname);\n+\t} else if (!(flags & REF_NO_REFLOG)) {\n+\t\tif (!new_sha1)\n+\t\t\tnew_sha1 = null_sha1;\n+\t\tif (log_ref_write(orig_refname, resolved_sha1,\n+\t\t\t\t new_sha1, msg, flags, err) < 0)\n+\t\t\tgoto done;\n+\t\tif (strcmp (refname, orig_refname) &&\n+\t\t log_ref_write(refname, resolved_sha1,\n+\t\t\t\t new_sha1, msg, flags, err) < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = 0;\n+done:\n+\tif (refname != orig_refname)\n+\t\tfree((void *) refname);\n+\treturn ret;\n+}\n+\n+static int lmdb_transaction_commit(struct ref_transaction *ref_transaction,\n+\t\t\t\t struct string_list *affected_refnames,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint ret = 0, i;\n+\tint n = ref_transaction->nr;\n+\tstruct ref_update **updates = ref_transaction->updates;\n+\n+\t/*\n+\t * We might already be in a write transaction, because some\n+\t * lmdb backend functionality is implemented in terms of\n+\t * (other stuff) + ref_transaction_commit\n+\t */\n+\tif (!in_write_transaction())\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tstruct ref_update *update = updates[i];\n+\n+\t\tif (lmdb_transaction_update(update->refname,\n+\t\t\t\t\t update->new_sha1,\n+\t\t\t\t\t (update->flags & REF_HAVE_OLD) ?\n+\t\t\t\t\t update->old_sha1 : NULL,\n+\t\t\t\t\t update->flags,\n+\t\t\t\t\t update->msg,\n+\t\t\t\t\t err)) {\n+\t\t\tmdb_txn_abort(transaction.txn);\n+\t\t\tret = -1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t}\n+\tret = mdb_transaction_commit(&transaction, err);\n+\n+cleanup:\n+\tref_transaction->state = REF_TRANSACTION_CLOSED;\n+\treturn ret;\n+}\n+\n+static int rename_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\n+\tconst char *newrefname = cb_data;\n+\tMDB_val key, new_key, val;\n+\n+\tassert(transaction.cursor);\n+\n+\tif (mdb_cursor_get_or_die(transaction.cursor, &key, &val, MDB_GET_CURRENT))\n+\t\tdie(\"renaming ref: mdb_cursor_get failed to get current\");\n+\n+\tnew_key.mv_size = strlen(newrefname) + 5 + 1 + 8;\n+\tnew_key.mv_data = xmalloc(new_key.mv_size);\n+\tstrcpy(new_key.mv_data, \"logs/\");\n+\tstrcpy((char *)new_key.mv_data + 5, newrefname);\n+\tmemcpy((char *)new_key.mv_data + new_key.mv_size - 8,\n+\t (const char *)key.mv_data + key.mv_size - 8, 8);\n+\tmdb_put_or_die(&transaction, &new_key, &val, 0);\n+\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\tfree(new_key.mv_data);\n+\treturn 0;\n+}\n+\n+static int lmdb_rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\tunsigned char orig_sha1[20];\n+\tint flag = 0;\n+\tint log = reflog_exists(oldref);\n+\tconst char *symref = NULL;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *ref_transaction;\n+\n+\tif (!strcmp(oldref, newref))\n+\t\treturn 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tref_transaction = ref_transaction_begin(&err);\n+\tif (!ref_transaction)\n+\t\tdie(\"%s\", err.buf);\n+\n+\tsymref = resolve_ref_unsafe(oldref, RESOLVE_REF_READING,\n+\t\t\t\t orig_sha1, &flag);\n+\tif (flag & REF_ISSYMREF) {\n+\t\terror(\"refname %s is a symbolic ref, renaming it is not supported\",\n+\t\t oldref);\n+\t\tgoto fail;\n+\t}\n+\tif (!symref) {\n+\t\tmdb_txn_abort(transaction.txn);\n+\t\terror(\"refname %s not found\", oldref);\n+\t\tgoto fail;\n+\t}\n+\tif (!rename_ref_available(oldref, newref))\n+\t\tgoto fail;\n+\n+\t/* Copy the reflog from the old to the new */\n+\tif (log) {\n+\t\tstruct strbuf old_log_sentinel = STRBUF_INIT;\n+\t\tMDB_val key;\n+\t\tint log_all;\n+\n+\t\tlog_all = log_all_ref_updates;\n+\t\tlog_all_ref_updates = 1;\n+\t\tif (safe_create_reflog(newref, 0, &err)) {\n+\t\t\terror(\"can't create reflog for %s: %s\", newref, err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\tgoto fail;\n+\t\t}\n+\t\tlog_all_ref_updates = log_all;\n+\n+\t\tfor_each_reflog_ent(oldref, rename_reflog_ent, (void *)newref);\n+\t\tstrbuf_addf(&old_log_sentinel, \"logs/%sxxxxxxxx\", oldref);\n+\t\tmemset(old_log_sentinel.buf + old_log_sentinel.len - 8, 0, 8);\n+\n+\t\tkey.mv_size = old_log_sentinel.len;\n+\t\tkey.mv_data = old_log_sentinel.buf;\n+\n+\t\t/* It's OK if the old reflog is missing */\n+\t\tmdb_del_or_die(&transaction, &key, NULL);\n+\t\tstrbuf_release(&old_log_sentinel);\n+\t}\n+\n+\tif (ref_transaction_delete(ref_transaction, oldref,\n+\t\t\t\t orig_sha1, REF_NODEREF, NULL, &err)) {\n+\t\terror(\"unable to delete old %s\", oldref);\n+\t\tgoto fail;\n+\t}\n+\n+\tif (ref_transaction_update(ref_transaction, newref, orig_sha1, NULL,\n+\t\t\t\t 0, logmsg, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tgoto fail;\n+\t}\n+\n+\tif (ref_transaction_commit(ref_transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tgoto fail;\n+\t}\n+\n+\treturn 0;\n+\n+fail:\n+\tref_transaction_free(ref_transaction);\n+\tstrbuf_release(&err);\n+\tmdb_txn_abort(transaction.txn);\n+\treturn 1;\n+}\n+\n+static int lmdb_delete_refs(struct string_list *refnames)\n+{\n+\tint i;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint result = 0;\n+\n+\tif (!refnames->nr)\n+\t\treturn 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tfor (i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[0].string;\n+\n+\t\tif (lmdb_transaction_update(refname, null_sha1, NULL,\n+\t\t\t\t\t 0, NULL, &err))\n+\t\t\tresult |= error(_(\"could not remove reference %s: %s\"),\n+\t\t\t\t\trefname, err.buf);\n+\t}\n+\n+\tresult |= mdb_transaction_commit(&transaction, &err);\n+\tstrbuf_release(&err);\n+\treturn 0;\n+}\n+\n+static int lmdb_for_each_reflog_ent_order(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data, int reverse)\n+{\n+\tMDB_val key, val;\n+\tchar *search_key;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum MDB_cursor_op direction = reverse ? MDB_PREV : MDB_NEXT;\n+\tuint64_t zero = 0ULL;\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsearch_key = xmalloc(len + 1);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\tstrcpy(search_key, log_path);\n+\n+\tif (reverse) {\n+\t\t/*\n+\t\t * For a reverse search, start at the key\n+\t\t * lexicographically after the searched-for key.\n+\t\t * That's the one with \\001 appended to the key.\n+\t\t */\n+\n+\t\tsearch_key[len - 1] = 1;\n+\t\tsearch_key[len] = 0;\n+\t\tkey.mv_size = len + 1;\n+\t} else {\n+\t\tkey.mv_size = len;\n+\t}\n+\n+\tkey.mv_data = search_key;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\ttransaction.cursor = cursor;\n+\n+\t/*\n+\t * MDB's cursor API requires that the first mdb_cursor_get be\n+\t * called with MDB_SET_RANGE. For reverse searches, this will\n+\t * give us the entry one-past the entry we're looking for, so\n+\t * we should jump back using MDB_PREV.\n+\t */\n+\tmdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\tif (direction == MDB_PREV)\n+\t\tmdb_cursor_get_or_die(cursor, &key, &val, direction);\n+\n+\tdo {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) || ((char *)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tif (!memcmp(&zero, ((char *)key.mv_data) + key.mv_size - 8, 8))\n+\t\t\tcontinue;\n+\n+\t\tassert(val.mv_size != 0);\n+\n+\t\tstrbuf_add(&sb, val.mv_data, val.mv_size - 1);\n+\t\tret = show_one_reflog_ent(&sb, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\n+\t\tstrbuf_reset(&sb);\n+\t} while (!mdb_cursor_get_or_die(cursor, &key, &val, direction));\n+\n+\tstrbuf_release(&sb);\n+\tfree(log_path);\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.for_each_reflog_ent(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 0);\n+}\n+\n+static int lmdb_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.for_each_reflog_ent_reverse(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 1);\n+}\n+\n+static int lmdb_reflog_exists(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 1;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.reflog_exists(refname);\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||\n+\t !starts_with(key.mv_data, log_path))\n+\t\tret = 0;\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\n+\treturn ret;\n+}\n+\n+struct wrapped_each_ref_fn {\n+\teach_ref_fn *fn;\n+\tvoid *cb_data;\n+};\n+\n+static int check_reflog(const char *refname,\n+\t\t\tconst struct object_id *oid, int flags, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn *wrapped = cb_data;\n+\n+\tif (reflog_exists(refname))\n+\t\treturn wrapped->fn(refname, oid, 0, wrapped->cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn wrapped = {fn, cb_data};\n+\tint result = head_ref(fn, cb_data);\n+\tif (result)\n+\t\treturn result;\n+\treturn for_each_ref(check_reflog, &wrapped);\n+}\n+\n+static int lmdb_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+{\n+\t/*\n+\t * We mark that there is a reflog by creating a key of the\n+\t * form logs/$refname followed by nine \\0 (one for\n+\t * string-termination, 8 in lieu of a timestamp), with an empty\n+\t * value.\n+\t */\n+\n+\tint in_transaction = in_write_transaction();\n+\tMDB_val key, val;\n+\n+\tif (!force_create && !should_autocreate_reflog(refname))\n+\t\treturn 0;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tkey.mv_size = strlen(refname) + 5 + 1 + 8;\n+\tkey.mv_data = xcalloc(1, key.mv_size);\n+\tsprintf((char *)key.mv_data, \"logs/%s\", refname);\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\n+\tfree(key.mv_data);\n+\tif (!in_transaction)\n+\t\treturn mdb_transaction_commit(&transaction, err);\n+\treturn 0;\n+}\n+\n+struct expire_reflog_cb {\n+\tunsigned int flags;\n+\treflog_expiry_should_prune_fn *should_prune_fn;\n+\tvoid *policy_cb;\n+\tunsigned char last_kept_sha1[20];\n+};\n+\n+static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\tstruct expire_reflog_cb *cb = cb_data;\n+\tstruct expire_reflog_policy_cb *policy_cb = cb->policy_cb;\n+\n+\tif (cb->flags & EXPIRE_REFLOGS_REWRITE)\n+\t\tosha1 = cb->last_kept_sha1;\n+\n+\tif ((*cb->should_prune_fn)(osha1, nsha1, email, timestamp, tz,\n+\t\t\t\t message, policy_cb)) {\n+\t\tif (cb->flags & EXPIRE_REFLOGS_DRY_RUN)\n+\t\t\tprintf(\"would prune %s\", message);\n+\t\telse {\n+\t\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\t\tprintf(\"prune %s\", message);\n+\n+\t\t\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\t\t}\n+\t} else {\n+\t\thashcpy(cb->last_kept_sha1, nsha1);\n+\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\tprintf(\"keep %s\", message);\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_ref(const char *refname, const unsigned char *sha1)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction;\n+\n+\ttransaction = ref_transaction_begin(&err);\n+\tif (!transaction) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (ref_transaction_update(transaction, refname, sha1, NULL,\n+\t\t\t\t REF_NO_REFLOG, NULL, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (ref_transaction_commit(transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int lmdb_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\tstruct expire_reflog_cb cb;\n+\tint dry_run = flags & EXPIRE_REFLOGS_DRY_RUN;\n+\tint status = 0;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char resolved_sha1[20];\n+\tint type;\n+\tchar *resolved;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.reflog_expire(refname, sha1, flags, prepare_fn,\n+\t\t\t\t\t should_prune_fn, cleanup_fn,\n+\t\t\t\t\t policy_cb_data);\n+\n+\tmemset(&cb, 0, sizeof(cb));\n+\tcb.flags = flags;\n+\tcb.policy_cb = policy_cb_data;\n+\tcb.should_prune_fn = should_prune_fn;\n+\n+\tlmdb_transaction_begin_flags_or_die(dry_run ? MDB_RDONLY : 0);\n+\n+\tresolved = check_ref(transaction.txn, refname, sha1,\n+\t\t\t resolved_sha1, 0, &type);\n+\tif (!resolved)\n+\t\tdie(\"Failed to resolve %s\", refname);\n+\tfree(resolved);\n+\n+\t(*prepare_fn)(refname, sha1, cb.policy_cb);\n+\tlmdb_for_each_reflog_ent(refname, expire_reflog_ent, &cb);\n+\t(*cleanup_fn)(cb.policy_cb);\n+\n+\tif (!dry_run) {\n+\t\t/*\n+\t\t * It doesn't make sense to adjust a reference pointed\n+\t\t * to by a symbolic ref based on expiring entries in\n+\t\t * the symbolic reference's reflog. Nor can we update\n+\t\t * a reference if there are no remaining reflog\n+\t\t * entries.\n+\t\t */\n+\t\tint update = (flags & EXPIRE_REFLOGS_UPDATE_REF) &&\n+\t\t\t!(type & REF_ISSYMREF) &&\n+\t\t\t!is_null_sha1(cb.last_kept_sha1);\n+\n+\t\tif (mdb_transaction_commit(&transaction, &err)) {\n+\t\t\tstatus |= error(\"couldn't write logs/%s: %s\", refname,\n+\t\t\t\t\terr.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t} else if (update &&\n+\t\t\t write_ref(refname, cb.last_kept_sha1)) {\n+\t\t\tstatus |= error(\"couldn't set %s\",\n+\t\t\t\t\trefname);\n+\t\t}\n+\t}\n+\treturn status;\n+}\n+\n+static int lmdb_pack_refs(unsigned int flags)\n+{\n+\t/* This concept does not exist in this backend. */\n+\treturn 0;\n+}\n+\n+static int lmdb_peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\tint flag;\n+\tunsigned char base[20];\n+\n+\tif (read_ref_full(refname, RESOLVE_REF_READING, base, &flag))\n+\t\treturn -1;\n+\n+\treturn peel_object(base, sha1);\n+}\n+\n+static int lmdb_create_symref(const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg)\n+{\n+\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char old_sha1[20], new_sha1[20];\n+\tMDB_val key, val;\n+\tchar *valdata;\n+\tint ret = 0;\n+\tint in_transaction;\n+\n+\tin_transaction = in_write_transaction();\n+\n+\tif (logmsg && read_ref(ref_target, old_sha1))\n+\t\thashclr(old_sha1);\n+\n+\tkey.mv_size = strlen(ref_target) + 1;\n+\tkey.mv_data = xstrdup(ref_target);\n+\n+\tval.mv_size = strlen(refs_heads_master) + 1 + 5;\n+\tvaldata = xmalloc(val.mv_size);\n+\tsprintf(valdata, \"ref: %s\", refs_heads_master);\n+\tval.mv_data = valdata;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\n+\tif (logmsg && !read_ref(refs_heads_master, new_sha1) &&\n+\t log_ref_write(ref_target, old_sha1, new_sha1, logmsg, 0, &err)) {\n+\t\terror(\"create_symref: log_ref_write failed: %s\", err.buf);\n+\t\tret = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (!in_transaction && mdb_transaction_commit(&transaction, &err)) {\n+\t\terror(\"create_symref: commit failed: %s\", err.buf);\n+\t\tret = -1;\n+\t}\n+\n+done:\n+\tstrbuf_release(&err);\n+\tfree(key.mv_data);\n+\tfree(valdata);\n+\n+\treturn ret;\n+}\n+\n+MDB_env *submodule_txn_begin(struct lmdb_transaction *transaction)\n+{\n+\tint ret;\n+\tMDB_env *submodule_env = NULL;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_git_path_submodule(&path, transaction->submodule, \"refdb\");\n+\n+\tif (!is_directory(path.buf))\n+\t\tgoto done;\n+\n+\tmkdir(path.buf, 0775);\n+\n+\tinit_env(&submodule_env, path.buf);\n+\n+\tret = mdb_txn_begin(submodule_env, NULL, MDB_RDONLY, &transaction->txn);\n+\tif (ret)\n+\t\tdie(\"mdb_txn_begin failed: %s\", mdb_strerror(ret));\n+\n+\tret = mdb_dbi_open(transaction->txn, NULL, 0, &transaction->dbi);\n+\tif (ret)\n+\t\tdie(\"mdb_txn_open failed: %s\", mdb_strerror(ret));\n+\n+done:\n+\tstrbuf_release(&path);\n+\treturn submodule_env;\n+}\n+\n+static int lmdb_resolve_gitlink_ref(const char *submodule, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n+{\n+\tstruct lmdb_transaction transaction;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\ttransaction.txn = NULL;\n+\ttransaction.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn -1;\n+\tresult = !resolve_ref_unsafe_txn(&transaction, refname,\n+\t\t\t\t\t RESOLVE_REF_READING, sha1, NULL);\n+\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result ? -1 : 0;\n+}\n+\n+static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (submodule) {\n+\t\tif (resolve_gitlink_ref(submodule, \"HEAD\", oid.hash) == 0)\n+\t\t\treturn fn(\"HEAD\", &oid, 0, cb_data);\n+\n+\t\treturn 0;\n+\t}\n+\n+\tif (!read_ref_full(\"HEAD\", RESOLVE_REF_READING, oid.hash, &flag))\n+\t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn do_head_ref(NULL, fn, cb_data);\n+}\n+\n+static int lmdb_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_head_ref(submodule, fn, cb_data);\n+}\n+\n+/*\n+ * Call fn for each reference for which the refname begins with base.\n+ * If trim is non-zero, then trim that many characters off the\n+ * beginning of each refname before passing the refname to fn. flags\n+ * can be DO_FOR_EACH_INCLUDE_BROKEN to include broken references in\n+ * the iteration. If fn ever returns a non-zero value, stop the\n+ * iteration and return that value; otherwise, return 0.\n+ */\n+static int do_for_each_ref(struct lmdb_transaction *transaction,\n+\t\t\t const char *base, each_ref_fn fn, int trim,\n+\t\t\t int flags, void *cb_data)\n+{\n+\n+\tMDB_val key, val;\n+\tMDB_cursor *cursor;\n+\tint baselen;\n+\tchar *search_key;\n+\tint retval;\n+\tint mdb_ret;\n+\n+\tretval = do_for_each_per_worktree_ref(transaction->submodule, base, fn,\n+\t\t\t\t\t trim, flags, cb_data);\n+\tif (retval)\n+\t\treturn retval;\n+\n+\tif (ref_paranoia < 0)\n+\t\tref_paranoia = git_env_bool(\"GIT_REF_PARANOIA\", 0);\n+\tif (ref_paranoia)\n+\t\tflags |= DO_FOR_EACH_INCLUDE_BROKEN;\n+\n+\tif (!base || !*base) {\n+\t\tbase = \"refs/\";\n+\t\ttrim = 0;\n+\t}\n+\n+\tbaselen = strlen(base);\n+\tsearch_key = xmalloc(baselen + 1);\n+\tstrcpy(search_key, base);\n+\tkey.mv_size = baselen + 1;\n+\tkey.mv_data = search_key;\n+\n+\tmdb_cursor_open_or_die(transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tstruct object_id oid;\n+\t\tint parsed_flags = 0;\n+\n+\t\tif (memcmp(key.mv_data, base, baselen))\n+\t\t\tbreak;\n+\n+\t\tparse_ref_data(transaction, (const char *)key.mv_data + (trim ? baselen : 0),\n+\t\t\t val.mv_data, oid.hash, 0, &parsed_flags, 0);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_BROKEN ||\n+\t\t (!(parsed_flags & REF_ISBROKEN) &&\n+\t\t has_sha1_file(oid.hash))) {\n+\t\t\tretval = fn((const char *)key.mv_data + (trim ? baselen : 0), &oid, parsed_flags, cb_data);\n+\t\t\tif (retval)\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\n+\treturn retval;\n+}\n+\n+static int lmdb_for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, \"\", fn, 0, 0, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct lmdb_transaction transaction;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref(fn, cb_data);\n+\n+\ttransaction.txn = NULL;\n+\ttransaction.submodule = submodule;\n+\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&transaction, \"\", fn, 0, 0, cb_data);\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, prefix, fn, strlen(prefix),\n+\t\t\t 0, cb_data);\n+}\n+\n+static int lmdb_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n+{\n+\tunsigned int flag = 0;\n+\n+\tif (broken)\n+\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, prefix, fn, 0, flag, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\tstruct lmdb_transaction transaction = {NULL};\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref_in(prefix, fn, cb_data);\n+\n+\ttransaction.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&transaction);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&transaction, prefix, fn,\n+\t\t\t\t strlen(prefix), 0, cb_data);\n+\tmdb_txn_abort(transaction.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, git_replace_ref_base, fn,\n+\t\t\t strlen(git_replace_ref_base), 0, cb_data);\n+}\n+\n+static int lmdb_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret;\n+\n+\tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tret = do_for_each_ref(&transaction, buf.buf, fn, 0, 0, cb_data);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction, \"\", fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_read(const char *key)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup;\n+\tint ret;\n+\tint needs_free = 0;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_get_or_die(&transaction, &key_val, &val, &needs_free);\n+\tfree(keydup);\n+\tswitch (ret) {\n+\tcase 0:\n+\t\tprintf(\"%s\\n\", (char *)val.mv_data);\n+\t\treturn 0;\n+\tcase MDB_NOTFOUND:\n+\t\tfprintf(stderr, \"%s not found\\n\", key);\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 2;\n+\t}\n+\tif (needs_free)\n+\t\tfree(val.mv_data);\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_write(const char *key, const char *value)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup, *valdup;\n+\n+\tif (ref_type(key) != REF_TYPE_NORMAL) {\n+\t\tval.mv_data = (void *)value;\n+\t\tval.mv_size = strlen(value) + 1;\n+\t\twrite_per_worktree_ref(NULL, key, &val);\n+\t\treturn;\n+\t}\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tvaldup = xstrdup(value);\n+\tval.mv_data = valdup;\n+\tval.mv_size = strlen(value) + 1;\n+\n+\tmdb_put_or_die(&transaction, &key_val, &val, 0);\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\n+\tfree(keydup);\n+\tfree(valdup);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_delete(const char *key)\n+{\n+\tMDB_val key_val;\n+\tchar *keydup;\n+\tint ret;\n+\n+\tif (ref_type(key) != REF_TYPE_NORMAL)\n+\t\treturn del_per_worktree_ref(NULL, key, NULL);\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_del_or_die(&transaction, &key_val, NULL);\n+\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\n+\tfree(keydup);\n+\treturn ret;\n+}\n+\n+static int print_raw_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t\tconst char *email, unsigned long timestamp,\n+\t\t\t\tint tz, const char *message, void *cb_data)\n+{\n+\tint *any = cb_data;\n+\t*any = 1;\n+\n+\tif (*message != '\\n')\n+\t\tprintf(\"%s %s %s %lu %+05d\\t%s\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz, message);\n+\telse\n+\t\tprintf(\"%s %s %s %lu %+05d\\n\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz);\n+\treturn 0;\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_reflog(const char *refname)\n+{\n+\tint any = 0;\n+\n+\tfor_each_reflog_ent(refname, print_raw_reflog_ent, &any);\n+\n+\treturn !any;\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_delete_reflog(char *refname)\n+{\n+\tMDB_val key, val;\n+\tint mdb_ret;\n+\tchar *search_key;\n+\tMDB_cursor *cursor;\n+\tint len;\n+\n+\tif (refname) {\n+\t\tlen = strlen(refname) + 5 + 1; /* logs/ + 0*/\n+\t\tsearch_key = xmalloc(len);\n+\t\tsprintf(search_key, \"logs/%s\", refname);\n+\t} else {\n+\t\tlen = 6; /* logs/ + 0*/\n+\t\tsearch_key = xstrdup(\"logs/\");\n+\t}\n+\tkey.mv_data = search_key;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\twhile (!mdb_ret) {\n+\t\tif (!starts_with(key.mv_data, search_key))\n+\t\t\tbreak;\n+\t\tif (refname && ((char *)val.mv_data)[len - 1] == 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\treturn;\n+}\n+\n+static void format_lmdb_reflog_ent(struct strbuf *dst, struct strbuf *src)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tconst char *msg;\n+\n+\tget_sha1_hex(src->buf, osha1);\n+\tget_sha1_hex(src->buf + 41, nsha1);\n+\n+\tmsg = strchr(src->buf + 82, '\\n');\n+\tassert(msg);\n+\tmsg += 1;\n+\n+\tformat_reflog_entry(dst, osha1, nsha1, src->buf + 82, msg);\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_append_reflog(const char *refname)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tuint64_t now = getnanotime();\n+\tMDB_val key, val;\n+\n+\tkey.mv_size = strlen(refname) + 14;\n+\tkey.mv_data = xcalloc(1, key.mv_size);\n+\tsprintf(key.mv_data, \"logs/%s\", refname);\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\t/* We do not remove the header here, because this is just for\n+\t * tests, so it's OK to be a bit inefficient */\n+\n+\twhile (strbuf_getwholeline(&input, stdin, '\\n') != EOF) {\n+\t\t/* \"logs/\" + \\0 + 8-byte timestamp for sorting and expiry */\n+\t\twrite_u64((char *)key.mv_data + key.mv_size - 8, htonll(now++));\n+\n+\t\t/*\n+\t\t * Convert the input from files-reflog format to\n+\t\t * lmdb-reflog-format\n+\t\t */\n+\n+\t\tformat_lmdb_reflog_ent(&sb, &input);\n+\t\tval.mv_data = sb.buf;\n+\t\tval.mv_size = sb.len + 1;\n+\t\tmdb_put_or_die(&transaction, &key, &val, 0);\n+\t\tstrbuf_reset(&sb);\n+\t\tinput.len = 0;\n+\t}\n+\n+\tstrbuf_release(&input);\n+\tstrbuf_release(&sb);\n+\tassert(mdb_transaction_commit(&transaction, NULL) == 0);\n+\tfree(key.mv_data);\n+}\n+\n+struct ref_be refs_be_lmdb = {\n+\tNULL,\n+\t\"lmdb\",\n+\tlmdb_init_backend,\n+\tlmdb_init_db,\n+\tlmdb_transaction_commit,\n+\tlmdb_transaction_commit, /* initial commit */\n+\n+\tlmdb_for_each_reflog_ent,\n+\tlmdb_for_each_reflog_ent_reverse,\n+\tlmdb_for_each_reflog,\n+\tlmdb_reflog_exists,\n+\tlmdb_create_reflog,\n+\tlmdb_delete_reflog,\n+\tlmdb_reflog_expire,\n+\n+\tlmdb_pack_refs,\n+\tlmdb_peel_ref,\n+\tlmdb_create_symref,\n+\tlmdb_delete_refs,\n+\tlmdb_rename_ref,\n+\n+\tlmdb_resolve_ref_unsafe,\n+\tlmdb_verify_refname_available,\n+\tlmdb_resolve_gitlink_ref,\n+\n+\tlmdb_head_ref,\n+\tlmdb_head_ref_submodule,\n+\tlmdb_for_each_ref,\n+\tlmdb_for_each_ref_submodule,\n+\tlmdb_for_each_ref_in,\n+\tlmdb_for_each_fullref_in,\n+\tlmdb_for_each_ref_in_submodule,\n+\tlmdb_for_each_rawref,\n+\tlmdb_for_each_namespaced_ref,\n+\tlmdb_for_each_replace_ref,\n+};\ndiff --git a/setup.c b/setup.c\nindex de6b8ac..9724c0b 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -279,10 +279,11 @@ int refdb_config(const char *var, const char *value, void *ptr)\n *\n * - either an objects/ directory _or_ the proper\n * GIT_OBJECT_DIRECTORY environment variable\n- * - a refs/ directory\n- * - either a HEAD symlink or a HEAD file that is formatted as\n- * a proper \"ref:\", or a regular file HEAD that has a properly\n- * formatted sha1 object name.\n+ * - a refdb/ directory or\n+ * - a refs/ directory\n+ * - either a HEAD symlink or a HEAD file that is formatted as\n+ * a proper \"ref:\", or a regular file HEAD that has a properly\n+ * formatted sha1 object name.\n */\n int is_git_directory(const char *suspect)\n {\n@@ -313,8 +314,13 @@ int is_git_directory(const char *suspect)\n \n \tstrbuf_setlen(&path, len);\n \tstrbuf_addstr(&path, \"/refs\");\n-\tif (access(path.buf, X_OK))\n-\t\tgoto done;\n+\n+\tif (access(path.buf, X_OK)) {\n+\t\tstrbuf_setlen(&path, len);\n+\t\tstrbuf_addstr(&path, \"/refdb\");\n+\t\tif (access(path.buf, X_OK))\n+\t\t\tgoto done;\n+\t}\n \n \tret = 1;\n done:\n@@ -383,6 +389,10 @@ static int check_repo_format(const char *var, const char *value, void *cb)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\trepository_format_precious_objects = git_config_bool(var, value);\n+#ifdef USE_LIBLMDB\n+\t\telse if (!strcmp(ext, \"refbackend\") && !strcmp(value, \"lmdb\"))\n+\t\t\t;\n+#endif\n \t\telse\n \t\t\tstring_list_append(&unknown_extensions, ext);\n \t}\ndiff --git a/test-refs-lmdb-backend.c b/test-refs-lmdb-backend.c\nnew file mode 100644\nindex 0000000..022cd21\n--- /dev/null\n+++ b/test-refs-lmdb-backend.c\n@@ -0,0 +1,68 @@\n+#include \"cache.h\"\n+#include \"string-list.h\"\n+#include \"parse-options.h\"\n+#include \"refs.h\"\n+\n+static const char * const test_refs_be_lmdb_usage[] = {\n+\t\"git test-refs-lmdb-backend \",\n+\t\"git test-refs-lmdb-backend \",\n+\tNULL,\n+};\n+\n+int test_refdb_raw_read(const char *key);\n+void test_refdb_raw_write(const char *key, const char *value);\n+int test_refdb_raw_reflog(const char *refname);\n+int test_refdb_raw_delete(const char *key);\n+void test_refdb_raw_delete_reflog(const char *refname);\n+void test_refdb_raw_append_reflog(const char *refname);\n+\n+int main(int argc, const char **argv)\n+{\n+\tconst char *delete = NULL;\n+\tconst char *reflog = NULL;\n+\tconst char *append_reflog = NULL;\n+\tint delete_missing_error = 0;\n+\tint clear_reflog = 0;\n+\tstruct refdb_config_data config_data = {NULL};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING('d', NULL, &delete, \"branch\", \"delete refdb entry\"),\n+\t\tOPT_STRING('l', NULL, &reflog, \"branch\", \"show reflog\"),\n+\t\tOPT_STRING('a', NULL, &append_reflog, \"branch\", \"append to reflog\"),\n+\t\tOPT_BOOL('c', NULL, &clear_reflog, \"delete reflog. If a branch is provided, the reflog for that branch will be deleted; else all reflogs will be deleted.\"),\n+\t\tOPT_BOOL('x', NULL, &delete_missing_error,\n+\t\t\t \"deleting a missing key is an error\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, \"\", options, test_refs_be_lmdb_usage,\n+\t\t\t 0);\n+\n+\tif (!append_reflog && !clear_reflog && !delete && !reflog && argc != 1 && argc != 2)\n+\t\tusage_with_options(test_refs_be_lmdb_usage,\n+\t\t\t\t options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tconfig_data.refs_backend_type = \"lmdb\";\n+\tconfig_data.refs_base = get_git_dir();\n+\n+\tregister_refs_backend(&refs_be_lmdb);\n+\tset_refs_backend(\"lmdb\", &config_data);\n+\n+\tif (clear_reflog) {\n+\t\ttest_refdb_raw_delete_reflog(argv[0]);\n+\t} else if (append_reflog) {\n+\t\ttest_refdb_raw_append_reflog(append_reflog);\n+\t} else if (reflog) {\n+\t\treturn test_refdb_raw_reflog(reflog);\n+\t} else if (delete) {\n+\t\tif (test_refdb_raw_delete(delete) && delete_missing_error)\n+\t\t\treturn 1;\n+\t} else if (argc == 1) {\n+\t\treturn test_refdb_raw_read(argv[0]);\n+\t} else {\n+\t\ttest_refdb_raw_write(argv[0], argv[1]);\n+\t}\n+\treturn 0;\n+}\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"273952","messageId":"1449102921-7707-17-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 16/16] refs: tests for lmdb backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-03T00:35:21Z","receivedAt":"2015-12-03T00:35:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add tests for the database backend.\n\nSigned-off-by: David Turner \nHelped-by: Dennis Kaarsemaker \n---\n t/t1460-refs-lmdb-backend.sh | 1109 +++++++++++++++++++++++++++++++++++\n t/t1470-refs-lmdb-backend-reflog.sh | 359 ++++++++++++\n t/test-lib.sh | 1 +\n 3 files changed, 1469 insertions(+)\n create mode 100755 t/t1460-refs-lmdb-backend.sh\n create mode 100755 t/t1470-refs-lmdb-backend-reflog.sh\n\ndiff --git a/t/t1460-refs-lmdb-backend.sh b/t/t1460-refs-lmdb-backend.sh\nnew file mode 100755\nindex 0000000..390e148\n--- /dev/null\n+++ b/t/t1460-refs-lmdb-backend.sh\n@@ -0,0 +1,1109 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2015 Twitter, Inc\n+# Copyright (c) 2006 Shawn Pearce\n+# This test is based on t1400-update-ref.sh\n+#\n+\n+test_description='Test lmdb refs backend'\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+if ! test_have_prereq LMDB\n+then\n+\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n+\ttest_done\n+fi\n+\n+raw_ref() {\n+\ttest-refs-lmdb-backend \"$1\"\n+}\n+\n+delete_ref() {\n+\ttest-refs-lmdb-backend -d \"$1\"\n+}\n+\n+write_ref() {\n+\ttest-refs-lmdb-backend \"$1\" \"$2\"\n+}\n+\n+raw_reflog() {\n+\ttest-refs-lmdb-backend -l \"$1\"\n+}\n+\n+delete_all_reflogs() {\n+\ttest-refs-lmdb-backend -c\n+}\n+\n+append_reflog() {\n+\ttest-refs-lmdb-backend -a \"$1\"\n+}\n+\n+Z=$_z40\n+\n+test_expect_success setup '\n+\tgit init --refs-backend-type=lmdb &&\n+\tfor name in A B C D E F\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tsha1=$(echo $name | git commit-tree $T) &&\n+\t\teval $name=$sha1\n+\tdone\n+'\n+\n+m=refs/heads/master\n+n_dir=refs/heads/gu\n+n=$n_dir/fixes\n+\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m with stale ref\" '\n+\ttest_must_fail git update-ref -d $m $A &&\n+\ttest $B = \"$(raw_ref $m)\"\n+'\n+test_expect_success \"delete $m\" '\n+\tgit update-ref -d $m $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \"delete $m without oldvalue verification\" \"\n+\tgit update-ref $m $A &&\n+\ttest $A = \\$(raw_ref $m) &&\n+\tgit update-ref -d $m &&\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"fail to create $n\" \\\n+\t\"git update-ref $n_dir $A &&\n+\t test_must_fail git update-ref $n $A >out 2>err\"\n+\n+delete_ref $n_dir\n+rm -f out err\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m (by HEAD) with stale ref\" '\n+\ttest_must_fail git update-ref -d HEAD $A &&\n+\ttest $B = $(raw_ref '\"$m\"')\n+'\n+test_expect_success \"delete $m (by HEAD)\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"pack refs\" \\\n+\t\"git pack-refs --all\"\n+test_expect_success \\\n+\t\"move $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"delete $m (by HEAD) should remove both packed and loose $m\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+OLD_HEAD=$(raw_ref HEAD)\n+test_expect_success \"delete symref without dereference\" '\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+\n+test_expect_success \"delete symref without dereference when the referred ref is packed\" '\n+\techo foo >foo.c &&\n+\tgit add foo.c &&\n+\tgit commit -m foo &&\n+\tgit pack-refs --all &&\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+delete_ref $m\n+\n+test_expect_success 'update-ref -d is not confused by self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\ttest_must_fail git update-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\tgit update-ref --no-deref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '\n+\ttest-refs-lmdb-backend refs/heads/bad \"\" &&\n+\ttest_when_finished \"delete_ref refs/heads/bad\" &&\n+\tgit symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&\n+\ttest_when_finished \"delete_ref refs/heads/ref-to-bad\" &&\n+\traw_ref refs/heads/ref-to-bad &&\n+\tgit update-ref --no-deref -d refs/heads/ref-to-bad &&\n+\t! raw_ref refs/heads/ref-to-bad\n+'\n+\n+test_expect_success '(not) create HEAD with old sha1' \"\n+\ttest_must_fail git update-ref HEAD $A $B\n+\"\n+test_expect_success \"(not) prior created .git/$m\" \"\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create HEAD\" \\\n+\t\"git update-ref HEAD $A\"\n+test_expect_success '(not) change HEAD with wrong SHA1' \"\n+\ttest_must_fail git update-ref HEAD $B $Z\n+\"\n+test_expect_success \"(not) changed .git/$m\" \"\n+\t! test $B\"' = $(raw_ref '\"$m\"')\n+'\n+\n+: a repository with working tree always has reflog these days...\n+delete_all_reflogs\n+: | append_reflog $m\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:30\" \\\n+\t git update-ref HEAD '\"$A\"' -m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:31\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:41\" \\\n+\t git update-ref HEAD'\" $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150200 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t\"raw_reflog $m >actual &&\n+\t test_cmp expect actual\"\n+delete_ref $m\n+delete_all_reflogs\n+: | append_reflog $m\n+rm -f actual expect\n+\n+test_expect_success \\\n+\t'enable core.logAllRefUpdates' \\\n+\t'git config core.logAllRefUpdates true &&\n+\t test true = $(git config --bool --get core.logAllRefUpdates)'\n+\n+test_expect_success \\\n+\t\"create $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:32\" \\\n+\t git update-ref HEAD'\" $A \"'-m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:33\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:43\" \\\n+\t git update-ref HEAD '\"$A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150320 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t'raw_reflog $m >actual &&\n+\ttest_cmp expect actual'\n+delete_ref $m\n+rm -f expect\n+\n+git update-ref $m $D\n+git reflog expire --expire=all $m\n+\n+append_reflog $m < 1117150320 -0500\n+$C $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150350 -0500\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 -0500\n+$F $Z $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150680 -0500\n+$Z $E $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 -0500\n+EOF\n+\n+ed=\"Thu, 26 May 2005 18:32:00 -0500\"\n+gd=\"Thu, 26 May 2005 18:33:00 -0500\"\n+ld=\"Thu, 26 May 2005 18:43:00 -0500\"\n+test_expect_success \\\n+\t'Query \"master@{May 25 2005}\" (before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 25 2005}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t\"Query master@{2005-05-25} (before history)\" \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify master@{2005-05-25} >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t echo test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:31:59}\" (1 second before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:31:59}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\''master'\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:00}\" (exactly history start)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:00}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:30}\" (first non-creation change)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:30}\" >o 2>e &&\n+\t test '\"$A\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:33:01}\" (middle of history with gap)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:33:01}\" >o 2>e &&\n+\t test '\"$B\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m has gap after $gd\"'.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:38:00}\" (middle of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:38:00}\" >o 2>e &&\n+\t test '\"$Z\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:43:00}\" (exact end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:43:00}\" >o 2>e &&\n+\t test '\"$E\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-28}\" (past end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-28}\" >o 2>e &&\n+\t test '\"$D\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m unexpectedly ended on $ld\"'.\" = \"$(cat e)\"'\n+\n+\n+git reflog expire --expire=all $m\n+delete_ref $m\n+\n+test_expect_success \\\n+ 'creating initial files' \\\n+ 'echo TEST >F &&\n+ git add F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:30\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:30\" git commit -m add -a &&\n+\t h_TEST=$(git rev-parse --verify HEAD) &&\n+\t echo The other day this did not work. >M &&\n+\t echo And then Bob told me how to fix it. >>M &&\n+\t echo OTHER >F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:41\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:41\" git commit -F M -a &&\n+\t h_OTHER=$(git rev-parse --verify HEAD) &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:44\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:44\" git commit --amend &&\n+\t h_FIXED=$(git rev-parse --verify HEAD) &&\n+\t echo Merged initial commit and a later commit. >M &&\n+\t echo $h_TEST >.git/MERGE_HEAD &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:45\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:45\" git commit -F M &&\n+\t h_MERGED=$(git rev-parse --verify HEAD) &&\n+\t rm -f M'\n+\n+cat >expect < 1117150200 +0000\tcommit (initial): add\n+$h_TEST $h_OTHER $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\tcommit: The other day this did not work.\n+$h_OTHER $h_FIXED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151040 +0000\tcommit (amend): The other day this did not work.\n+$h_FIXED $h_MERGED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151100 +0000\tcommit (merge): Merged initial commit and a later commit.\n+EOF\n+test_expect_success \\\n+\t'git commit logged updates' \\\n+\t\"raw_reflog $m >actual &&\n+\ttest_cmp expect actual\"\n+unset h_TEST h_OTHER h_FIXED h_MERGED\n+\n+test_expect_success \\\n+\t'git cat-file blob master:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob master:F)'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:30}:F (expect TEST)' \\\n+\t'test TEST = $(git cat-file blob \"master@{2005-05-26 23:30}:F\")'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:42}:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob \"master@{2005-05-26 23:42}:F\")'\n+\n+a=refs/heads/a\n+b=refs/heads/b\n+c=refs/heads/c\n+E='\"\"'\n+F='%s\\0'\n+pws='path with space'\n+\n+test_expect_success 'stdin test setup' '\n+\techo \"$pws\" >\"$pws\" &&\n+\tgit add -- \"$pws\" &&\n+\tgit commit -m \"$pws\"\n+'\n+\n+test_expect_success '-z fails without --stdin' '\n+\ttest_must_fail git update-ref -z $m $m $m 2>err &&\n+\tgrep \"usage: git update-ref\" err\n+'\n+\n+test_expect_success 'stdin works with no input' '\n+\t>stdin &&\n+\tgit update-ref --stdin stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin fails on only whitespace' '\n+\techo \" \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin fails on leading whitespace' '\n+\techo \" create $a $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a $m\" err\n+'\n+\n+test_expect_success 'stdin fails on unknown command' '\n+\techo \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin fails on unbalanced quotes' '\n+\techo \"create $a \\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails on invalid escape' '\n+\techo \"create $a \\\"ma\\zter\\\"\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"ma\\\\\\\\zter\\\\\\\"\" err\n+'\n+\n+test_expect_success 'stdin fails on junk after quoted argument' '\n+\techo \"create \\\"$a\\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unexpected character after quoted argument: \\\\\\\"$a\\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails create with no ref' '\n+\techo \"create \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with no new value' '\n+\techo \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with too many arguments' '\n+\techo \"create $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails update with no ref' '\n+\techo \"update \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with no new value' '\n+\techo \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with too many arguments' '\n+\techo \"update $a $m $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails delete with no ref' '\n+\techo \"delete \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin fails delete with too many arguments' '\n+\techo \"delete $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails verify with too many arguments' '\n+\techo \"verify $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: verify $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails option with unknown name' '\n+\techo \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin fails with duplicate refs' '\n+\tcat >stdin <<-EOF &&\n+\tcreate $a $m\n+\tcreate $b $m\n+\tcreate $a $m\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin create ref works' '\n+\techo \"create $a $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with quoted argument' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"$m\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with escaped character' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"ma\\\\163ter\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update ref creates with zero old value' '\n+\techo \"update $b $m $Z\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin update ref creates with empty old value' '\n+\techo \"update $b $m $E\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin create ref works with path with space to blob' '\n+\techo \"create refs/blobs/pws \\\"$m:$pws\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin update ref fails with wrong old value' '\n+\techo \"update $c $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref fails with bad old value' '\n+\techo \"update $c $m does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with bad new value' '\n+\techo \"create $c does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with zero new value' '\n+\techo \"create $c \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: zero \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref works with right old value' '\n+\techo \"update $b $m~1 $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with wrong old value' '\n+\techo \"delete $a $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with zero old value' '\n+\techo \"delete $a \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tupdate refs/TESTSYMREF $a $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tdelete refs/TESTSYMREF $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref works with right old value' '\n+\techo \"delete $b $m~1\" >stdin &&\n+\tgit update-ref --stdin stdin <<-EOF &&\n+\tupdate $a $m\n+\tcreate $b $m\n+\tverify $c\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $m\" >stdin &&\n+\tgit update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify succeeds for missing reference' '\n+\techo \"verify refs/heads/missing $Z\" >stdin &&\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref --stdin expect &&\n+\techo \"verify $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $Z\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update refs works with identity updates' '\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c $Z $E\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c ''\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tcat >stdin <<-EOF &&\n+\tdelete $a $m\n+\tupdate $b $Z $m\n+\tupdate $c $E $m~1\n+\tEOF\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on empty command' '\n+\tprintf $F \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin -z fails on only whitespace' '\n+\tprintf $F \" \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on leading whitespace' '\n+\tprintf $F \" create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails on unknown command' '\n+\tprintf $F \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no ref' '\n+\tprintf $F \"create \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no new value' '\n+\tprintf $F \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with too many arguments' '\n+\tprintf $F \"create $a\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no ref' '\n+\tprintf $F \"update \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too few args' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z emits warning with empty new value' '\n+\tgit update-ref $a $m &&\n+\tprintf $F \"update $a\" \"\" \"\" >stdin &&\n+\tgit update-ref -z --stdin err &&\n+\tgrep \"warning: update $a: missing , treating as zero\" err &&\n+\ttest_must_fail git rev-parse --verify -q $a\n+'\n+\n+test_expect_success 'stdin -z fails update with no new value' '\n+\tprintf $F \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no old value' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too many arguments' '\n+\tprintf $F \"update $m\" \"$m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no ref' '\n+\tprintf $F \"delete \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no old value' '\n+\tprintf $F \"delete $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with too many arguments' '\n+\tprintf $F \"delete $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with too many arguments' '\n+\tprintf $F \"verify $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with no old value' '\n+\tprintf $F \"verify $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: verify $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails option with unknown name' '\n+\tprintf $F \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin -z fails with duplicate refs' '\n+\tprintf $F \"create $a\" \"$m\" \"create $b\" \"$m\" \"create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin -z create ref works' '\n+\tprintf $F \"create $a\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update ref creates with zero old value' '\n+\tprintf $F \"update $b\" \"$m\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin -z update ref creates with empty old value' '\n+\tprintf $F \"update $b\" \"$m\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref works with path with space to blob' '\n+\tprintf $F \"create refs/blobs/pws\" \"$m:$pws\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin -z update ref fails with wrong old value' '\n+\tprintf $F \"update $c\" \"$m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref fails with bad old value' '\n+\tprintf $F \"update $c\" \"$m\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails when ref exists' '\n+\tgit update-ref $c $m &&\n+\tgit rev-parse \"$c\" >expect &&\n+\tprintf $F \"create $c\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse \"$c\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref fails with bad new value' '\n+\tgit update-ref -d \"$c\" &&\n+\tprintf $F \"create $c\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails with empty new value' '\n+\tprintf $F \"create $c\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: missing \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref works with right old value' '\n+\tprintf $F \"update $b\" \"$m~1\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with wrong old value' '\n+\tprintf $F \"delete $a\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with zero old value' '\n+\tprintf $F \"delete $a\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"update refs/TESTSYMREF\" \"$a\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"delete refs/TESTSYMREF\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref works with right old value' '\n+\tprintf $F \"delete $b\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for missing reference' '\n+\tprintf $F \"verify refs/heads/missing\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tprintf $F \"verify $m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update refs works with identity updates' '\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$Z\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tprintf $F \"delete $a\" \"$m\" \"update $b\" \"$Z\" \"$m\" \"update $c\" \"\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin /dev/null || test-refs-lmdb-backend -l \"$1\"\n+}\n+\n+append_reflog() {\n+\ttest-refs-lmdb-backend -a \"$1\"\n+}\n+\n+check_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\" && git cat-file -t $o || {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone &&\n+\ttest -z \"$gaah\"\n+}\n+\n+check_fsck () {\n+\toutput=$(git fsck --full)\n+\tcase \"$1\" in\n+\t'')\n+\t\ttest -z \"$output\" ;;\n+\t*)\n+\t\techo \"$output\" | grep \"$1\" ;;\n+\tesac\n+}\n+\n+corrupt () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmv .git/objects/$aa/$zz .git/$aa$zz\n+}\n+\n+recover () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmkdir -p .git/objects/$aa\n+\tmv .git/$aa$zz .git/objects/$aa/$zz\n+}\n+\n+check_dont_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\"\n+\t\tgit cat-file -t $o && {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone\n+\ttest -z \"$gaah\"\n+}\n+\n+test_expect_success setup '\n+\tgit init --refs-backend-type=lmdb &&\n+\tmkdir -p A/B &&\n+\techo rat >C &&\n+\techo ox >A/D &&\n+\techo tiger >A/B/E &&\n+\tgit add . &&\n+\n+\ttest_tick && git commit -m rabbit &&\n+\tH=`git rev-parse --verify HEAD` &&\n+\tA=`git rev-parse --verify HEAD:A` &&\n+\tB=`git rev-parse --verify HEAD:A/B` &&\n+\tC=`git rev-parse --verify HEAD:C` &&\n+\tD=`git rev-parse --verify HEAD:A/D` &&\n+\tE=`git rev-parse --verify HEAD:A/B/E` &&\n+\tcheck_fsck &&\n+\n+\ttest_chmod +x C &&\n+\tgit add C &&\n+\ttest_tick && git commit -m dragon &&\n+\tL=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f C A/B/E &&\n+\techo snake >F &&\n+\techo horse >A/G &&\n+\tgit add F A/G &&\n+\ttest_tick && git commit -a -m sheep &&\n+\tF=`git rev-parse --verify HEAD:F` &&\n+\tG=`git rev-parse --verify HEAD:A/G` &&\n+\tI=`git rev-parse --verify HEAD:A` &&\n+\tJ=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f A/G &&\n+\ttest_tick && git commit -a -m monkey &&\n+\tK=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tcheck_fsck &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success rewind '\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\ttest -f C &&\n+\ttest -f A/B/E &&\n+\t! test -f F &&\n+\t! test -f A/G &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog\n+'\n+\n+test_expect_success 'corrupt and check' '\n+\n+\tcorrupt $F &&\n+\tcheck_fsck \"missing blob $F\"\n+\n+'\n+\n+test_expect_success 'reflog expire --dry-run should not touch reflog' '\n+\n+\tgit reflog expire --dry-run \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog &&\n+\n+\tcheck_fsck \"missing blob $F\"\n+'\n+\n+test_expect_success 'reflog expire' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\techo git reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 2 reflog &&\n+\n+\tcheck_fsck \"dangling commit $K\"\n+'\n+\n+test_expect_success 'prune and fsck' '\n+\n+\tgit prune &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E H L &&\n+\tcheck_dont_have F G I J K\n+\n+'\n+\n+test_expect_success 'recover and check' '\n+\n+\trecover $F &&\n+\tcheck_fsck \"dangling blob $F\"\n+\n+'\n+\n+test_expect_success 'delete' '\n+\techo 1 > C &&\n+\ttest_tick &&\n+\tgit commit -m rat C &&\n+\n+\techo 2 > C &&\n+\ttest_tick &&\n+\tgit commit -m ox C &&\n+\n+\techo 3 > C &&\n+\ttest_tick &&\n+\tgit commit -m tiger C &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\tmaster_entry_count=$(git reflog show master | wc -l) &&\n+\n+\ttest $HEAD_entry_count = 5 &&\n+\ttest $master_entry_count = 5 &&\n+\n+\n+\tgit reflog delete master@{1} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\ttest $HEAD_entry_count = $(git reflog | wc -l) &&\n+\t! grep ox < output &&\n+\n+\tmaster_entry_count=$(wc -l < output) &&\n+\n+\tgit reflog delete HEAD@{1} &&\n+\ttest $(($HEAD_entry_count -1)) = $(git reflog | wc -l) &&\n+\ttest $master_entry_count = $(git reflog show master | wc -l) &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\n+\tgit reflog delete master@{07.04.2005.15:15:00.-0700} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\t! grep dragon < output\n+\n+'\n+\n+test_expect_success 'rewind2' '\n+\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success '--expire=never' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=never \\\n+\t\t--expire-unreachable=never \\\n+\t\t--all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=never' '\n+\n+\tgit config gc.reflogexpire never &&\n+\tgit config gc.reflogexpireunreachable never &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=false' '\n+\n+\tgit config gc.reflogexpire false &&\n+\tgit config gc.reflogexpireunreachable false &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog &&\n+\n+\tgit config --unset gc.reflogexpire &&\n+\tgit config --unset gc.reflogexpireunreachable\n+\n+'\n+\n+test_expect_success 'checkout should not delete log for packed ref' '\n+\ttest $(git reflog master | wc -l) = 4 &&\n+\tgit branch foo &&\n+\tgit pack-refs --all &&\n+\tgit checkout foo &&\n+\ttest $(git reflog master | wc -l) = 4\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs on)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# now logs/refs/heads/one is a stale directory, but\n+\t# we should move it out of the way to create \"one\" reflog\n+\tgit branch one master &&\n+\techo \"one@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs off)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# same as before, but we only create a reflog for \"one\" if\n+\t# it already exists, which it does not\n+\tgit -c core.logallrefupdates=false branch one master &&\n+\t: >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Triggering the bug detected by this test requires a newline to fall\n+# exactly BUFSIZ-1 bytes from the end of the file. We don't know\n+# what that value is, since it's platform dependent. However, if\n+# we choose some value N, we also catch any D which divides N evenly\n+# (since we will read backwards in chunks of D). So we choose 8K,\n+# which catches glibc (with an 8K BUFSIZ) and *BSD (1K).\n+#\n+# Each line is 114 characters, so we need 75 to still have a few before the\n+# last 8K. The 89-character padding on the final entry lines up our\n+# newline exactly.\n+test_expect_success 'parsing reverse reflogs at BUFSIZ boundaries' '\n+\tgit checkout -b reflogskip &&\n+\tz38=00000000000000000000000000000000000000 &&\n+\tident=\"abc 0000000001 +0000\" &&\n+\tfor i in $(test_seq 1 75); do\n+\t\tprintf \"$z38%02d $z38%02d %s\\t\" $i $(($i+1)) \"$ident\" &&\n+\t\tif test $i = 75; then\n+\t\t\tfor j in $(test_seq 1 89); do\n+\t\t\t\tprintf X\n+\t\t\tdone\n+\t\telse\n+\t\t\tprintf X\n+\t\tfi &&\n+\t\tprintf \"\\n\"\n+\tdone | append_reflog refs/heads/reflogskip &&\n+\tgit rev-parse reflogskip@{73} >actual &&\n+\techo ${z38}03 >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 16c4d7b..c6ba541 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -958,6 +958,7 @@ test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE\" && test_set_prereq LIBPCRE\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$USE_LIBLMDB\" && test_set_prereq LMDB\n \n # Can we rely on git's output in the C locale?\n if test -n \"$GETTEXT_POISON\"\n-- \n2.4.2.749.g0ed01d8-twtrsrc\n"},{"id":"274040","messageId":"xmqqegf1pxll.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-2-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 01/16] refs: add a backend method structure with transaction functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-05T00:07:02Z","receivedAt":"2015-12-05T00:07:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> diff --git a/refs.c b/refs.c\n> index 0f7628d..babba8a 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -10,6 +10,31 @@\n> #include \"tag.h\"\n> \n> /*\n> + * We always have a files backend and it is the default.\n> + */\n> +extern struct ref_be refs_be_files;\n\nIt is customary to s/extern //; in C sources.\n"},{"id":"274041","messageId":"xmqq7fktpxlf.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-14-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-05T00:07:08Z","receivedAt":"2015-12-05T00:07:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> diff --git a/setup.c b/setup.c\n> index d343725..de6b8ac 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1,5 +1,6 @@\n> #include \"cache.h\"\n> #include \"dir.h\"\n> +#include \"refs.h\"\n> #include \"string-list.h\"\n> \n> static int inside_git_dir = -1;\n> @@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n> \treturn ret;\n> }\n> \n> +int refdb_config(const char *var, const char *value, void *ptr)\n> +{\n> + struct refdb_config_data *cdata = ptr;\n> +\n> + if (!strcmp(var, \"core.refsbackendtype\"))\n> +\t cdata->refs_backend_type = xstrdup((char *)value);\n> + return 0;\n> +}\n> +\n\nThese lines seem to be indented with SPs not HTs.\n\n> /*\n> * Test if it looks like we're at a git directory.\n> * We want to see:\n"},{"id":"274042","messageId":"xmqqzixpoiyt.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-16-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-05T00:08:26Z","receivedAt":"2015-12-05T00:08:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> +\twhile (!mdb_ret) {\n> +\t\tif (starts_with(key.mv_data, refname) &&\n> +\t\t ((char*)key.mv_data)[refname_len - 2] == '/') {\n\nERROR: \"(foo*)\" should be \"(foo *)\"\n#877: FILE: refs/lmdb-backend.c:514:\n+ ((char*)key.mv_data)[refname_len - 2] == '/') {\n\n> +static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)\n> +{\n> +\tunsigned char osha1[20], nsha1[20];\n> +\tchar *email_end, *message;\n> +\tunsigned long timestamp;\n> +\tint tz;\n> +\n> +\t/* old (raw) new (raw) name SP time TAB msg LF */\n> +\tif (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n> +\t !(email_end = strchr(sb->buf + 40, '>')) ||\n> +\t email_end[1] != ' ' ||\n> +\t !(timestamp = strtoul(email_end + 2, &message, 10)) ||\n> +\t !message || message[0] != ' ' ||\n> +\t (message[1] != '+' && message[1] != '-') ||\n> +\t !isdigit(message[2]) || !isdigit(message[3]) ||\n> +\t !isdigit(message[4]) || !isdigit(message[5]))\n> +\t\treturn 0; /* corrupt? */\n\nERROR: do not use assignment in if condition\n#1024: FILE: refs/lmdb-backend.c:661:\n+ if (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n\n> +static char *check_ref(MDB_txn *txn, const char *refname,\n> +\t\t const unsigned char *old_sha1,\n> +\t\t unsigned char *resolved_sha1, int flags,\n> +\t\t int *type_p)\n> +{\n> +\tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n> +\tint resolve_flags = 0;\n> +\tint type;\n> +\tchar *resolved_refname;\n> +\n> +\tif (mustexist)\n> +\t\tresolve_flags |= RESOLVE_REF_READING;\n> +\tif (flags & REF_DELETING) {\n> +\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n> +\t\tif (flags & REF_NODEREF)\n> +\t\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n> +\t}\n> +\n> +\t/*\n> +\t * The first time we resolve the refname, we're just trying to\n> +\t * see if there is any ref at all by this name, even if it is\n> +\t * a broken symref.\n> +\t */\n> +\trefname = resolve_ref_unsafe(refname, resolve_flags,\n> +\t\t\t\t resolved_sha1, &type);\n> +\tif (type_p)\n> +\t *type_p = type;\n\nWARNING: suspect code indent for conditional statements (8, 12)\n#1177: FILE: refs/lmdb-backend.c:814:\n+ if (type_p)\n+ *type_p = type;\n\nIndeed, this line should be indented with two HTs.\n\n> +\twhile (!mdb_ret) {\n> +\t\tif (key.mv_size < len)\n> +\t\t\tbreak;\n> +\n> +\t\tif (!starts_with(key.mv_data, log_path) || ((char*)key.mv_data)[len - 1] != 0)\n\nERROR: \"(foo*)\" should be \"(foo *)\"\n#1254: FILE: refs/lmdb-backend.c:891:\n+ if (!starts_with(key.mv_data, log_path) || ((char*)key.mv_data)[len - 1] != 0)\n\n> +\t\tif (strcmp (refname, orig_refname) &&\n\nWARNING: space prohibited between function name and open parenthesis '('\n#1366: FILE: refs/lmdb-backend.c:1003:\n+ if (strcmp (refname, orig_refname) &&\n\n\nThis concludes my first pass, mechanical \"lint\" (with help from\ncheckpatch.pl). I'll hopefully have time to do the more meaningful\ndesign and implementation review later.\n\nThanks.\n"},{"id":"274044","messageId":"1449275136.3858.2.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqzixpoiyt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-05T00:25:36Z","receivedAt":"2015-12-05T00:25:36Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-12-04 at 16:08 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > +\twhile (!mdb_ret) {\n> > +\t\tif (starts_with(key.mv_data, refname) &&\n> > +\t\t ((char*)key.mv_data)[refname_len - 2] == '/') {\n> \n> ERROR: \"(foo*)\" should be \"(foo *)\"\n> #877: FILE: refs/lmdb-backend.c:514:\n> + ((char*)key.mv_data)[refname_len - 2] == '/') {\n> \n> > +static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)\n> > +{\n> > +\tunsigned char osha1[20], nsha1[20];\n> > +\tchar *email_end, *message;\n> > +\tunsigned long timestamp;\n> > +\tint tz;\n> > +\n> > +\t/* old (raw) new (raw) name SP time TAB msg LF */\n> > +\tif (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n> > +\t !(email_end = strchr(sb->buf + 40, '>')) ||\n> > +\t email_end[1] != ' ' ||\n> > +\t !(timestamp = strtoul(email_end + 2, &message, 10)) ||\n> > +\t !message || message[0] != ' ' ||\n> > +\t (message[1] != '+' && message[1] != '-') ||\n> > +\t !isdigit(message[2]) || !isdigit(message[3]) ||\n> > +\t !isdigit(message[4]) || !isdigit(message[5]))\n> > +\t\treturn 0; /* corrupt? */\n> \n> ERROR: do not use assignment in if condition\n> #1024: FILE: refs/lmdb-backend.c:661:\n> + if (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n\nThis code is based on code from files-backend.c. But I can change it\nanyway.\n\nI'll also fix the rest, but wait for further review to post.\n"},{"id":"274047","messageId":"CACsJy8DDKW4np7N+KA=dpz9uNke0+cyQD-J3U74VM=4WbsjrKQ@mail.gmail.com","threadId":"40926","inReplyTo":"1449102921-7707-14-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-12-05T06:30:13Z","receivedAt":"2015-12-05T06:30:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Dec 3, 2015 at 1:35 AM, David Turner wrote:\n> git init learns a new argument --refs-backend-type. Presently, only\n> \"files\" is supported, but later we will add other backends.\n>\n> When this argument is used, the repository's core.refsBackendType\n> configuration value is set, and the refs backend's initdb function is\n> used to set up the ref database.\n\ngit-init can also be used on existing repos. What happens in that case\nif we use --refs-backend-type. Will existing refs be migrated to the\nnew backend or hidden away?\n-- \nDuy\n"},{"id":"274051","messageId":"20151205074444.GD21639@sigill.intra.peff.net","threadId":"40926","inReplyTo":"CACsJy8DDKW4np7N+KA=dpz9uNke0+cyQD-J3U74VM=4WbsjrKQ@mail.gmail.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-05T07:44:44Z","receivedAt":"2015-12-05T07:44:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Dec 05, 2015 at 07:30:13AM +0100, Duy Nguyen wrote:\n\n> On Thu, Dec 3, 2015 at 1:35 AM, David Turner wrote:\n> > git init learns a new argument --refs-backend-type. Presently, only\n> > \"files\" is supported, but later we will add other backends.\n> >\n> > When this argument is used, the repository's core.refsBackendType\n> > configuration value is set, and the refs backend's initdb function is\n> > used to set up the ref database.\n> \n> git-init can also be used on existing repos. What happens in that case\n> if we use --refs-backend-type. Will existing refs be migrated to the\n> new backend or hidden away?\n\nIt would be neat if it migrated, but I suspect that may introduce\ncomplications. It's probably OK in the initial implementation to bail if\nthe option is used in an existing repo.\n\nI think the config option needs to be extensions.refsBackendType, too,\nper the logic in 00a09d5 (introduce \"extensions\" form of\ncore.repositoryformatversion, 2015-06-23). And I guess it needs to bump\ncore.repositoryformatversion to \"1\".\n\n-Peff\n"},{"id":"274160","messageId":"1449535132.21906.0.camel@twopensource.com","threadId":"40926","inReplyTo":"20151205074444.GD21639@sigill.intra.peff.net","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-08T00:38:52Z","receivedAt":"2015-12-08T00:38:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2015-12-05 at 02:44 -0500, Jeff King wrote:\n> On Sat, Dec 05, 2015 at 07:30:13AM +0100, Duy Nguyen wrote:\n> \n> > On Thu, Dec 3, 2015 at 1:35 AM, David Turner wrote:\n> > > git init learns a new argument --refs-backend-type. Presently, only\n> > > \"files\" is supported, but later we will add other backends.\n> > >\n> > > When this argument is used, the repository's core.refsBackendType\n> > > configuration value is set, and the refs backend's initdb function is\n> > > used to set up the ref database.\n> > \n> > git-init can also be used on existing repos. What happens in that case\n> > if we use --refs-backend-type. Will existing refs be migrated to the\n> > new backend or hidden away?\n\nI'd like to migrate, but I don't want to write that code prematurely. I\nfigure we can get everything else right first. \n\n> It would be neat if it migrated, but I suspect that may introduce\n> complications. It's probably OK in the initial implementation to bail if\n> the option is used in an existing repo.\n\nWill fix.\n\n> I think the config option needs to be extensions.refsBackendType, too,\n> per the logic in 00a09d5 (introduce \"extensions\" form of\n> core.repositoryformatversion, 2015-06-23). And I guess it needs to bump\n> core.repositoryformatversion to \"1\".\n\nI did that in a later patch, but now is a better time; will fix.\n"},{"id":"274233","messageId":"20151210180213.GA27152@sigill.intra.peff.net","threadId":"40926","inReplyTo":"1449102921-7707-14-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-10T18:02:13Z","receivedAt":"2015-12-10T18:02:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 02, 2015 at 07:35:18PM -0500, David Turner wrote:\n\n> @@ -510,9 +511,36 @@ int validate_headref(const char *path)\n> \tunsigned char sha1[20];\n> \tint fd;\n> \tssize_t len;\n> +\tstruct refdb_config_data refdb_data = {NULL, NULL};\n> +\n> +\tif (lstat(path, &st) < 0) {\n> +\t\tint backend_type_set;\n> +\t\tstruct strbuf config_path = STRBUF_INIT;\n> +\t\tif (path) {\n> +\t\t\tchar *pathdup = xstrdup(path);\n> +\t\t\tchar *git_dir = dirname(pathdup);\n> +\t\t\tstrbuf_addf(&config_path, \"%s/%s\", git_dir, \"config\");\n> +\t\t\tfree(pathdup);\n> +\t\t} else {\n> +\t\t\tstrbuf_addstr(&config_path, \"config\");\n> +\t\t}\n\nCan we ever follow this \"else\" branch? It would mean feeding NULL to\nlstat.\n\n-Peff\n"},{"id":"274242","messageId":"1449776161.21906.1.camel@twopensource.com","threadId":"40926","inReplyTo":"20151210180213.GA27152@sigill.intra.peff.net","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-10T19:36:01Z","receivedAt":"2015-12-10T19:36:01Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-12-10 at 13:02 -0500, Jeff King wrote:\n> On Wed, Dec 02, 2015 at 07:35:18PM -0500, David Turner wrote:\n> \n> > @@ -510,9 +511,36 @@ int validate_headref(const char *path)\n> > \tunsigned char sha1[20];\n> > \tint fd;\n> > \tssize_t len;\n> > +\tstruct refdb_config_data refdb_data = {NULL, NULL};\n> > +\n> > +\tif (lstat(path, &st) < 0) {\n> > +\t\tint backend_type_set;\n> > +\t\tstruct strbuf config_path = STRBUF_INIT;\n> > +\t\tif (path) {\n> > +\t\t\tchar *pathdup = xstrdup(path);\n> > +\t\t\tchar *git_dir = dirname(pathdup);\n> > +\t\t\tstrbuf_addf(&config_path, \"%s/%s\", git_dir, \"config\");\n> > +\t\t\tfree(pathdup);\n> > +\t\t} else {\n> > +\t\t\tstrbuf_addstr(&config_path, \"config\");\n> > +\t\t}\n> \n> Can we ever follow this \"else\" branch? It would mean feeding NULL to\n> lstat.\n\nProbably cruft from an old version of the patch. Removed if/else.\nThanks.\n"},{"id":"274300","messageId":"xmqqr3isd0hj.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-3-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-11T23:33:28Z","receivedAt":"2015-12-11T23:33:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> diff --git a/cache.h b/cache.h\n> index 51c35c3..707455a 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1111,6 +1111,13 @@ extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as s\n> extern int interpret_branch_name(const char *str, int len, struct strbuf *);\n> extern int get_sha1_mb(const char *str, unsigned char *sha1);\n> \n> +/*\n> + * Return true iff abbrev_name is a possible abbreviation for\n> + * full_name according to the rules defined by ref_rev_parse_rules in\n> + * refs.c.\n> + */\n> +extern int refname_match(const char *abbrev_name, const char *full_name);\n> +\n> extern int validate_headref(const char *ref);\n\nThis hunk is strange. It duplicates what is in refs.h, risking the\ntwo copies to drift away from each other over time.\n\nAs the only callsites are in remote.c that includes refs.h, this\nprobably should be dropped.\n\nOther changes that give redirection to various operations via vtable\nlooked all sensible.\n\n> struct ref_be {\n> \tstruct ref_be *next;\n> \tconst char *name;\n> \tref_transaction_commit_fn *transaction_commit;\n> +\n> +\tpack_refs_fn *pack_refs;\n> +\tpeel_ref_fn *peel_ref;\n> +\tcreate_symref_fn *create_symref;\n> +\n> +\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n> +\tverify_refname_available_fn *verify_refname_available;\n> +\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n> };\n> \n> #endif /* REFS_REFS_INTERNAL_H */\n"},{"id":"274302","messageId":"xmqqmvtgd06p.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-3-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-11T23:39:58Z","receivedAt":"2015-12-11T23:39:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> struct ref_be {\n> \tstruct ref_be *next;\n> \tconst char *name;\n> \tref_transaction_commit_fn *transaction_commit;\n> +\n> +\tpack_refs_fn *pack_refs;\n> +\tpeel_ref_fn *peel_ref;\n> +\tcreate_symref_fn *create_symref;\n> +\n> +\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n> +\tverify_refname_available_fn *verify_refname_available;\n> +\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n> };\n\nThis may have been pointed out in the previous reviews by somebody\nelse, but I think it is more customary to declare a struct member\nthat is a pointer to a customization function without leading '*',\ni.e.\n\n\ttypedef TYPE (*customize_fn)(ARGS);\n\n struct vtable {\n\t\t...\n \tcutomize_fn fn;\n\t\t...\n\t};\n\nin our codebase (cf. string_list::cmp, prio_queue::compare).\n"},{"id":"274303","messageId":"1449877765.1678.2.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqmvtgd06p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-11T23:49:25Z","receivedAt":"2015-12-11T23:49:25Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-12-11 at 15:39 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > struct ref_be {\n> > \tstruct ref_be *next;\n> > \tconst char *name;\n> > \tref_transaction_commit_fn *transaction_commit;\n> > +\n> > +\tpack_refs_fn *pack_refs;\n> > +\tpeel_ref_fn *peel_ref;\n> > +\tcreate_symref_fn *create_symref;\n> > +\n> > +\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n> > +\tverify_refname_available_fn *verify_refname_available;\n> > +\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n> > };\n> \n> This may have been pointed out in the previous reviews by somebody\n> else, but I think it is more customary to declare a struct member\n> that is a pointer to a customization function without leading '*',\n> i.e.\n> \n> \ttypedef TYPE (*customize_fn)(ARGS);\n> \n> struct vtable {\n> \t\t...\n> \tcutomize_fn fn;\n> \t\t...\n> \t};\n> \n> in our codebase (cf. string_list::cmp, prio_queue::compare).\n\nThe previous review was here:\nhttp://permalink.gmane.org/gmane.comp.version-control.git/279062\n\nMichael wrote:\n> Hmmm, I thought our convention was to define typedefs for functions\n> themselves, not for the pointer-to-function; e.g.,\n>\n> typedef struct ref_transaction *ref_transaction_begin_fn(struct\n> strbuf *err);\n>\n> (which would require `struct ref_be` to be changed to\n>\n> ref_transaction_begin_fn *transaction_begin;\n\nAnd you agreed. So I changed it. Do you want me to change it back? \n"},{"id":"274304","messageId":"1449877797.1678.3.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqr3isd0hj.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-11T23:49:57Z","receivedAt":"2015-12-11T23:49:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-12-11 at 15:33 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > diff --git a/cache.h b/cache.h\n> > index 51c35c3..707455a 100644\n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -1111,6 +1111,13 @@ extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as s\n> > extern int interpret_branch_name(const char *str, int len, struct strbuf *);\n> > extern int get_sha1_mb(const char *str, unsigned char *sha1);\n> > \n> > +/*\n> > + * Return true iff abbrev_name is a possible abbreviation for\n> > + * full_name according to the rules defined by ref_rev_parse_rules in\n> > + * refs.c.\n> > + */\n> > +extern int refname_match(const char *abbrev_name, const char *full_name);\n> > +\n> > extern int validate_headref(const char *ref);\n> \n> This hunk is strange. It duplicates what is in refs.h, risking the\n> two copies to drift away from each other over time.\n> \n> As the only callsites are in remote.c that includes refs.h, this\n> probably should be dropped.\n\nWill fix, thanks.\n"},{"id":"274305","messageId":"xmqqio44czm1.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449102921-7707-5-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 04/16] refs: add do_for_each_per_worktree_ref","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-11T23:52:22Z","receivedAt":"2015-12-11T23:52:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> Alternate refs backends might still use files to store per-worktree\n> refs. So the files backend's ref-loading infrastructure should be\n> available to those backends, just for use on per-worktree refs. Add\n> do_for_each_per_worktree_ref, which iterates over per-worktree refs.\n\nIs this \"might still use\"? I am instead getting an impression that,\nwe are declaring that the recommended way to store per-worktree refs\nis with filesystem backend.\n\nNot complaining against either such a design decision (if that is\nwhat is made here) or the above description in the log message--just\nwant to understand the intention better.\n\nI also wonder if it is cleaner to have a single interface that lets\nanybody ask a pointer to the backend struct by name. With such an\ninterface, a new backend that wants to delegate per-worktree refs to\nfiles backend could\n\n\tstatic int DB_for_each_ref(...)\n {\n\t\tstruct ref_be *files = locate_ref_backend(\"files\");\n\t\tfiles->for_each_ref(... | WORKTREE_ONLY, ...);\n\t\t... do its own enumeration of non-per-worktree refs ...\n\t}\n\nand such a delegation does not need to be limited to per-worktree\niteration.\n\nOr is that a road to expose too much implementation details of\na backend to other backends?\n\n> Signed-off-by: David Turner \n> ---\n> refs/files-backend.c | 15 ++++++++++++---\n> refs/refs-internal.h | 10 ++++++++++\n> 2 files changed, 22 insertions(+), 3 deletions(-)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index d4bd6cf..bde4892 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -518,9 +518,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n> \tdir->sorted = dir->nr = i;\n> }\n> \n> -/* Include broken references in a do_for_each_ref*() iteration: */\n> -#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n> -\n> /*\n> * Return true iff the reference described by entry can be resolved to\n> * an object in the database. Emit a warning if the referred-to\n> @@ -568,6 +565,10 @@ static int do_one_ref(struct ref_entry *entry, void *cb_data)\n> \tstruct ref_entry *old_current_ref;\n> \tint retval;\n> \n> +\tif (data->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n> +\t ref_type(entry->name) != REF_TYPE_PER_WORKTREE)\n> +\t\treturn 0;\n> +\n> \tif (!starts_with(entry->name, data->base))\n> \t\treturn 0;\n> \n> @@ -1738,6 +1739,14 @@ static int do_for_each_ref(struct ref_cache *refs, const char *base,\n> \treturn do_for_each_entry(refs, base, do_one_ref, &data);\n> }\n> \n> +int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n> +\t\t\t\t each_ref_fn fn, int trim, int flags,\n> +\t\t\t\t void *cb_data)\n> +{\n> +\treturn do_for_each_ref(get_ref_cache(submodule), base, fn, trim,\n> +\t\t\t flags | DO_FOR_EACH_PER_WORKTREE_ONLY, cb_data);\n> +}\n> +\n> static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n> {\n> \tstruct object_id oid;\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index ad683df..433d0fe 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -42,6 +42,16 @@\n> * value to ref_update::flags\n> */\n> \n> +/* Include broken references in a do_for_each_ref*() iteration */\n> +#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n> +\n> +/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n> +#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n> +\n> +int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n> +\t\t\t\t each_ref_fn fn, int trim, int flags,\n> +\t\t\t\t void *cb_data);\n> +\n> /*\n> * Return true iff refname is minimally safe. \"Safe\" here means that\n> * deleting a loose reference by this name will not do any damage, for\n"},{"id":"274306","messageId":"1449878510.1678.7.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqio44czm1.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/16] refs: add do_for_each_per_worktree_ref","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-12T00:01:50Z","receivedAt":"2015-12-12T00:01:50Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-12-11 at 15:52 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > Alternate refs backends might still use files to store per-worktree\n> > refs. So the files backend's ref-loading infrastructure should be\n> > available to those backends, just for use on per-worktree refs. Add\n> > do_for_each_per_worktree_ref, which iterates over per-worktree refs.\n> \n> Is this \"might still use\"? I am instead getting an impression that,\n> we are declaring that the recommended way to store per-worktree refs\n> is with filesystem backend.\n> \n> Not complaining against either such a design decision (if that is\n> what is made here) or the above description in the log message--just\n> want to understand the intention better.\n\nI'm not super-happy with the notion of splitting data across backends;\nit's a bit messy. I've resigned myself to it because it seems like the\nsimplest way forward in the presence of per-worktree refs. But I\nfigured the message could leave it open in case we change our minds.\n\n> I also wonder if it is cleaner to have a single interface that lets\n> anybody ask a pointer to the backend struct by name. With such an\n> interface, a new backend that wants to delegate per-worktree refs to\n> files backend could\n> \n> \tstatic int DB_for_each_ref(...)\n> {\n> \t\tstruct ref_be *files = locate_ref_backend(\"files\");\n> \t\tfiles->for_each_ref(... | WORKTREE_ONLY, ...);\n> \t\t... do its own enumeration of non-per-worktree refs ...\n> \t}\n> \n> and such a delegation does not need to be limited to per-worktree\n> iteration.\n> \n> Or is that a road to expose too much implementation details of\n> a backend to other backends?\n\nGiven that the files backend is special in terms of its rights and\nresponsibilities, I think it's OK to just refer to it directly. I think\nallowing backends to generally pick and choose bits from other backends\nwould be confusing.\n"},{"id":"274308","messageId":"xmqqegescy6y.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1449877765.1678.2.camel@twopensource.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-12T00:23:01Z","receivedAt":"2015-12-12T00:23:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> The previous review was here:\n> http://permalink.gmane.org/gmane.comp.version-control.git/279062\n>\n> Michael wrote:\n>> Hmmm, I thought our convention was to define typedefs for functions\n>> themselves, not for the pointer-to-function; e.g.,\n>>\n>> typedef struct ref_transaction *ref_transaction_begin_fn(struct\n>> strbuf *err);\n>>\n>> (which would require `struct ref_be` to be changed to\n>>\n>> ref_transaction_begin_fn *transaction_begin;\n>\n> And you agreed. So I changed it. Do you want me to change it back? \n\nSorry about that. I was agreeing to my mistaken understanding of\nwhat was pointed out X-<.\n\nWe do prefer fn(args) over (*fn)(args) and somehow I mixed that up\nwith the declaration style.\n"},{"id":"274309","messageId":"1449881317.1678.8.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqegescy6y.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-12T00:48:37Z","receivedAt":"2015-12-12T00:48:37Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-12-11 at 16:23 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > The previous review was here:\n> > http://permalink.gmane.org/gmane.comp.version-control.git/279062\n> >\n> > Michael wrote:\n> >> Hmmm, I thought our convention was to define typedefs for functions\n> >> themselves, not for the pointer-to-function; e.g.,\n> >>\n> >> typedef struct ref_transaction *ref_transaction_begin_fn(struct\n> >> strbuf *err);\n> >>\n> >> (which would require `struct ref_be` to be changed to\n> >>\n> >> ref_transaction_begin_fn *transaction_begin;\n> >\n> > And you agreed. So I changed it. Do you want me to change it back? \n> \n> Sorry about that. I was agreeing to my mistaken understanding of\n> what was pointed out X-<.\n> \n> We do prefer fn(args) over (*fn)(args) and somehow I mixed that up\n> with the declaration style.\n\nSo I should change it back?\n"},{"id":"274645","messageId":"20151217010013.GC6594@google.com","threadId":"40926","inReplyTo":"1449102921-7707-16-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-12-17T01:00:13Z","receivedAt":"2015-12-17T01:00:13Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"David Turner wrote:\n\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -496,6 +496,13 @@ core.repositoryFormatVersion::\n> \tInternal variable identifying the repository format and layout\n> \tversion.\n> \n> +core.refsBackendType::\n> +\tType of refs backend. Default is to use the original files\n> +\tbased backend. Set to 'lmdb' to activate the lmdb database\n> +\tbackend. If you use the lmdb backend,\n> +\tcore.repositoryFormatVersion must be set to 1, and\n> +\textensions.refBackend must be set to 'lmdb'.\n\nIf core.refsBackendType names one backend and extensions.refBackend\nnames another, which wins?\n\nThanks,\nJonathan\n"},{"id":"274646","messageId":"1450319481.11927.0.camel@twopensource.com","threadId":"40926","inReplyTo":"20151217010013.GC6594@google.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-17T02:31:21Z","receivedAt":"2015-12-17T02:31:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-16 at 17:00 -0800, Jonathan Nieder wrote:\n> David Turner wrote:\n> \n> > --- a/Documentation/config.txt\n> > +++ b/Documentation/config.txt\n> > @@ -496,6 +496,13 @@ core.repositoryFormatVersion::\n> > \tInternal variable identifying the repository format and\n> > layout\n> > \tversion.\n> > \n> > +core.refsBackendType::\n> > +\tType of refs backend. Default is to use the original files\n> > +\tbased backend. Set to 'lmdb' to activate the lmdb database\n> > +\tbackend. If you use the lmdb backend,\n> > +\tcore.repositoryFormatVersion must be set to 1, and\n> > +\textensions.refBackend must be set to 'lmdb'.\n> \n> If core.refsBackendType names one backend and extensions.refBackend\n> names another, which wins?\n\nI've simplified this to just use extensions.refsBackendType for\neverything. Will re-roll once I finish getting comments on the rest of\nthis series.\n"},{"id":"274690","messageId":"20151217204952.GE6594@google.com","threadId":"40926","inReplyTo":"1450319481.11927.0.camel@twopensource.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-12-17T20:49:53Z","receivedAt":"2015-12-17T20:49:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"David Turner wrote:\n> On Wed, 2015-12-16 at 17:00 -0800, Jonathan Nieder wrote:\n>> David Turner wrote:\n\n>>> +core.refsBackendType::\n>>> +\tType of refs backend. Default is to use the original files\n>>> +\tbased backend. Set to 'lmdb' to activate the lmdb database\n>>> +\tbackend. If you use the lmdb backend,\n>>> +\tcore.repositoryFormatVersion must be set to 1, and\n>>> +\textensions.refBackend must be set to 'lmdb'.\n>>\n>> If core.refsBackendType names one backend and extensions.refBackend\n>> names another, which wins?\n>\n> I've simplified this to just use extensions.refsBackendType for\n> everything. Will re-roll once I finish getting comments on the rest of\n> this series.\n\nPerfect. Thanks for the quick reply.\n\nJonathan\n"},{"id":"274714","messageId":"loom.20151218T050438-482@post.gmane.org","threadId":"40926","inReplyTo":"xmqqmvtgd06p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/16] refs: add methods for misc ref operations","fromName":"Howard Chu","fromEmail":"hyc@symas.com","sentAt":"2015-12-18T04:06:20Z","receivedAt":"2015-12-18T04:06:20Z","isPatch":true,"sender":{"key":"hyc@symas.com","avatar":"https://gravatar.com/avatar/07931cfff11bc93bbafb5090ff38d3e0dda8745df2e0913457d29ead5d400462?d=mp&s=160"},"body":"Junio C Hamano pobox.com> writes:\n\n> \n> David Turner twopensource.com> writes:\n> \n> > struct ref_be {\n> > \tstruct ref_be *next;\n> > \tconst char *name;\n> > \tref_transaction_commit_fn *transaction_commit;\n> > +\n> > +\tpack_refs_fn *pack_refs;\n> > +\tpeel_ref_fn *peel_ref;\n> > +\tcreate_symref_fn *create_symref;\n> > +\n> > +\tresolve_ref_unsafe_fn *resolve_ref_unsafe;\n> > +\tverify_refname_available_fn *verify_refname_available;\n> > +\tresolve_gitlink_ref_fn *resolve_gitlink_ref;\n> > };\n> \n> This may have been pointed out in the previous reviews by somebody\n> else, but I think it is more customary to declare a struct member\n> that is a pointer to a customization function without leading '*',\n> i.e.\n> \n> \ttypedef TYPE (*customize_fn)(ARGS);\n> \n> struct vtable {\n> \t\t...\n> \tcutomize_fn fn;\n> \t\t...\n> \t};\n> \n> in our codebase (cf. string_list::cmp, prio_queue::compare).\n\n(LMDB author here, just passing by)\n\nIMO you're making a mistake. You should always typedef the thing itself, not\na pointer to the thing. If you only typedef the pointer, you can only use\nthe typedef to declare pointers and never the thing itself. If you typedef\nthe actual thing, you can use the typedef to declare both the thing and\npointer to thing.\n\nIt's particularly useful to have function typedefs because later on you can\nuse the actual typedef to declare instances of that function type in header\nfiles, and guarantee that your function definitions match what they're\nintended to match. Otherwise, assigning a bare (*func) to something that\nmismatches only generates a warning, instead of an error.\n\n-- \n -- Howard Chu\n CTO, Symas Corp. http://www.symas.com\n Director, Highland Sun http://highlandsun.com/hyc/\n Chief Architect, OpenLDAP http://www.openldap.org/project/ \n"},{"id":"274926","messageId":"1450828577.3892.12.camel@twopensource.com","threadId":"40926","inReplyTo":"1449102921-7707-1-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 00/16] LMDB refs backend atop pre-vtable","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-22T23:56:17Z","receivedAt":"2015-12-22T23:56:17Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-02 at 19:35 -0500, David Turner wrote:\n> I'm starting the patchset numbering over from 1 here, because this\n> version of the patchset is a subset of the last version.\n\nI'm going out of town on 12/24 and will return 1/5. I won't be reading\nthis email address. When I get back, I'll read any reviews.\n"},{"id":"274931","messageId":"567A3228.2060407@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-9-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 08/16] refs: add methods to init refs backend and db","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T05:33:28Z","receivedAt":"2015-12-23T05:33:28Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> Alternate refs backends might not need the refs/heads directory and so\n> on, so we make ref db initialization part of the backend. We also\n> might need to initialize ref backends themselves, so we'll add a\n> method for that as well.\n> \n> Signed-off-by: David Turner \n> ---\n> builtin/init-db.c | 14 ++++----------\n> refs.c | 8 +++++++-\n> refs.h | 4 +++-\n> refs/files-backend.c | 23 +++++++++++++++++++++++\n> refs/refs-internal.h | 4 ++++\n> 5 files changed, 41 insertions(+), 12 deletions(-)\n> \n> [...]\n> diff --git a/refs.c b/refs.c\n> index 9a2fed7..bdeb276 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -22,13 +22,14 @@ struct ref_be *refs_backends = &refs_be_files;\n> /*\n> * This function is used to switch to an alternate backend.\n> */\n> -int set_refs_backend(const char *name)\n> +int set_refs_backend(const char *name, void *data)\n\nThe purpose of the data argument is rather mysterious at this point,\nespecially since set_refs_backend() still doesn't have any callers. I\nhope that will become clearer later in the series. Nevertheless, it\nwould be nice for its use to be described in the docstring (which should\npreferably be moved to the header file).\n\n> [...]\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index e769242..6600c02 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3313,6 +3313,11 @@ static int ref_present(const char *refname,\n> \treturn string_list_has_string(affected_refnames, refname);\n> }\n> \n> +void files_init_backend(void *data)\n> +{\n> +\t/* do nothing */\n> +}\n> +\n> static int files_initial_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t\t\t struct strbuf *err)\n> {\n> @@ -3534,9 +3539,27 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n> \treturn -1;\n> }\n> \n> +static int files_init_db(struct strbuf *err, int shared)\n> +{\n> +\t/*\n> +\t * Create .git/refs/{heads,tags}\n> +\t */\n> +\tsafe_create_dir(git_path(\"refs\"), 1);\n> +\tsafe_create_dir(git_path(\"refs/heads\"), 1);\n> +\tsafe_create_dir(git_path(\"refs/tags\"), 1);\n> +\tif (shared) {\n> +\t\tadjust_shared_perm(git_path(\"refs\"));\n> +\t\tadjust_shared_perm(git_path(\"refs/heads\"));\n> +\t\tadjust_shared_perm(git_path(\"refs/tags\"));\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> struct ref_be refs_be_files = {\n> \tNULL,\n> \t\"files\",\n> +\tfiles_init_backend,\n> +\tfiles_init_db,\n> \tfiles_transaction_commit,\n> \tfiles_initial_transaction_commit,\n> \n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 478ad54..85a0b91 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -208,6 +208,8 @@ const char *find_descendant_ref(const char *dirname,\n> int rename_ref_available(const char *oldname, const char *newname);\n> \n> /* refs backends */\n> +typedef void ref_backend_init_fn(void *data);\n> +typedef int ref_backend_init_db_fn(struct strbuf *err, int shared);\n> typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,\n> \t\t\t\t struct strbuf *err);\n> \n> @@ -267,6 +269,8 @@ typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);\n> struct ref_be {\n> \tstruct ref_be *next;\n> \tconst char *name;\n> +\tref_backend_init_fn *init_backend;\n> +\tref_backend_init_db_fn *init_db;\n> \tref_transaction_commit_fn *transaction_commit;\n> \tref_transaction_commit_fn *initial_transaction_commit;\n> \n> \n\nYour naming seems inconsistent here. I would have expected a\n\"files_init_backend()\" function to satisfy the typedef\n\"ref_backend_init_backend_fn\" or \"ref_init_backend_fn\", not\n\"ref_backend_init_fn\". Or, conversely, for the function implementing\n\"ref_backend_init_fn\" to be called \"files_init()\".\n\nMore generally, it would be nice to have a consistent naming pattern\nbetween (1) the name of the typedef, (2) the name of the member in\n\"struct ref_be\", (3) the names of concrete, backend-specific\nimplementations of the functions.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274932","messageId":"567A3EB6.9000405@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-12-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 11/16] refs: move duplicate check to common code","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T06:27:02Z","receivedAt":"2015-12-23T06:27:02Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> The check for duplicate refnames in a transaction is needed for\n> all backends, so move it to the common code.\n> \n> ref_transaction_commit_fn gains a new argument, the sorted\n> string_list of affected refnames.\n> \n> Signed-off-by: David Turner \n> ---\n> refs.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n> refs/files-backend.c | 57 ++++-------------------------------------\n> refs/refs-internal.h | 1 +\n> 3 files changed, 75 insertions(+), 54 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 1b79630..808053f 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1093,6 +1093,37 @@ const char *find_descendant_ref(const char *dirname,\n> \treturn NULL;\n> }\n> \n> +/*\n> + * Return 1 if there are any duplicate refnames in the updates in\n> + * `transaction`, and fill in err with an appropriate error message.\n> + * Fill in `refnames` with the refnames from the transaction.\n> + */\n> +\n> +static int ref_update_reject_duplicates(struct ref_transaction *transaction,\n> +\t\t\t\t\tstruct string_list *refnames,\n> +\t\t\t\t\tstruct strbuf *err)\n\nI like that you extract this code into a function. Though it feels\nawkward to have a function called \"ref_update_reject_duplicates()\" that\nhas a side effect of filling the names into a string list. I think it\nwould feel more natural to call the function get_affected_refnames(),\nand treat the duplicate check as an extra bonus.\n\nYou could go even farther and split it into two functions,\n\nvoid get_affected_refnames(struct ref_transaction *transaction,\n struct string_list *refnames);\nint ref_update_reject_duplicates(struct string_list *refnames,\n struct strbuf *err);\n\n> +{\n> +\tint i, n = transaction->nr;\n> +\tstruct ref_update **updates;\n> +\n> +\tassert(err);\n> +\n> +\tupdates = transaction->updates;\n> +\t/* Fail if a refname appears more than once in the transaction: */\n> +\tfor (i = 0; i < n; i++)\n> +\t\tstring_list_append(refnames, updates[i]->refname);\n> +\tstring_list_sort(refnames);\n> +\n> +\tfor (i = 1; i < n; i++)\n> +\t\tif (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {\n> +\t\t\tstrbuf_addf(err,\n> +\t\t\t\t \"Multiple updates for ref '%s' not allowed.\",\n> +\t\t\t\t refnames->items[i].string);\n> +\t\t\treturn 1;\n> +\t\t}\n> +\treturn 0;\n> +}\n> +\n> /* backend functions */\n> int refs_init_db(struct strbuf *err, int shared)\n> {\n> @@ -1102,7 +1133,29 @@ int refs_init_db(struct strbuf *err, int shared)\n> int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t struct strbuf *err)\n> {\n> -\treturn the_refs_backend->transaction_commit(transaction, err);\n> +\tint ret = -1;\n> +\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\n> +\tassert(err);\n> +\n> +\tif (transaction->state != REF_TRANSACTION_OPEN)\n> +\t\tdie(\"BUG: commit called for transaction that is not open\");\n> +\n> +\tif (!transaction->nr) {\n> +\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tif (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {\n> +\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tret = the_refs_backend->transaction_commit(transaction,\n> +\t\t\t\t\t\t &affected_refnames, err);\n> +done:\n> +\tstring_list_clear(&affected_refnames, 0);\n> +\treturn ret;\n> }\n\nHere you are avoiding a small amount of code duplication by calling\nref_update_reject_duplicates() here rather than in the backend-specific\ncode. But you are doing so at the cost of having to compute\naffected_refnames here and pass it (redundantly) to the backend's\ntransaction_commit function. This increases the coupling between these\nfunctions, which in my opinion is worse than the small amount of code\nduplication. But maybe it's just me.\n\nThe check of transaction->state, on the other hand, makes sense here.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274933","messageId":"1450853697.3892.21.camel@twopensource.com","threadId":"40926","inReplyTo":"567A3228.2060407@alum.mit.edu","subject":"Re: [PATCH 08/16] refs: add methods to init refs backend and db","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-12-23T06:54:57Z","receivedAt":"2015-12-23T06:54:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 06:33 +0100, Michael Haggerty wrote:\n> On 12/03/2015 01:35 AM, David Turner wrote:\n> > Alternate refs backends might not need the refs/heads directory and\n> > so\n> > on, so we make ref db initialization part of the backend. We also\n> > might need to initialize ref backends themselves, so we'll add a\n> > method for that as well.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > builtin/init-db.c | 14 ++++----------\n> > refs.c | 8 +++++++-\n> > refs.h | 4 +++-\n> > refs/files-backend.c | 23 +++++++++++++++++++++++\n> > refs/refs-internal.h | 4 ++++\n> > 5 files changed, 41 insertions(+), 12 deletions(-)\n> > \n> > [...]\n> > diff --git a/refs.c b/refs.c\n> > index 9a2fed7..bdeb276 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -22,13 +22,14 @@ struct ref_be *refs_backends = &refs_be_files;\n> > /*\n> > * This function is used to switch to an alternate backend.\n> > */\n> > -int set_refs_backend(const char *name)\n> > +int set_refs_backend(const char *name, void *data)\n> \n> The purpose of the data argument is rather mysterious at this point,\n> especially since set_refs_backend() still doesn't have any callers. I\n> hope that will become clearer later in the series. Nevertheless, it\n> would be nice for its use to be described in the docstring (which\n> should\n> preferably be moved to the header file).\n\nWill fix.\n\n> > struct ref_be {\n> > \tstruct ref_be *next;\n> > \tconst char *name;\n> > +\tref_backend_init_fn *init_backend;\n> > +\tref_backend_init_db_fn *init_db;\n> > \tref_transaction_commit_fn *transaction_commit;\n> > \tref_transaction_commit_fn *initial_transaction_commit;\n> > \n> > \n> \n> Your naming seems inconsistent here. I would have expected a\n> \"files_init_backend()\" function to satisfy the typedef\n> \"ref_backend_init_backend_fn\" or \"ref_init_backend_fn\", not\n> \"ref_backend_init_fn\". Or, conversely, for the function implementing\n> \"ref_backend_init_fn\" to be called \"files_init()\".\n> \n> More generally, it would be nice to have a consistent naming pattern\n> between (1) the name of the typedef, (2) the name of the member in\n> \"struct ref_be\", (3) the names of concrete, backend-specific\n> implementations of the functions.\n\nI'll change this to \"refs_init_backend_fn *init_backend\" and\n\"refs_init_db_fn *init_db\" (since we already have an init_db function),\nand make any other similar changes I happen to notice.\n\nOn the naming in general, I am somewhat at the mercy of history here. \n For example, git presently has functions named ref_transaction_commit\n(object_verb) and create_reflog (verb_object). So nothing I do will be\nconsistent with everything. I could, of course, do some initial\ncommits to clean up the naming, but that would create code churn.\n"},{"id":"274935","messageId":"567A5516.9070209@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-13-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 12/16] refs: always handle non-normal refs in files backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T08:02:30Z","receivedAt":"2015-12-23T08:02:30Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> Always handle non-normal (per-worktree or pseudo) refs in the files\n> backend instead of alternate backends.\n> \n> Sometimes a ref transaction will update both a per-worktree ref and a\n> normal ref. For instance, an ordinary commit might update\n> refs/heads/master and HEAD (or at least HEAD's reflog).\n> \n> We handle three cases here:\n> \n> 1. updates to normal refs continue to go through the chosen backend\n> \n> 2. updates to non-normal refs with REF_NODEREF or to non-symbolic refs\n> are moved to a separate files backend transaction.\n> \n> 3. updates to symbolic refs are dereferenced to their base ref. The\n> update to the base ref then goes through the ordinary backend, while\n> the files backend is directly called to update the symref's reflog.\n> \n> Signed-off-by: David Turner \n> ---\n> refs.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 139 insertions(+), 2 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 808053f..e48e43a 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -9,6 +9,11 @@\n> #include \"object.h\"\n> #include \"tag.h\"\n> \n> +const char split_transaction_fail_warning[] =\n> +\t\"A ref transaction was split across two refs backends. Part of the \"\n> +\t\"transaction succeeded, but then the update to the per-worktree refs \"\n> +\t\"failed. Your repository may be in an inconsistent state.\";\n> +\n> /*\n> * We always have a files backend and it is the default.\n> */\n> @@ -784,6 +789,13 @@ void ref_transaction_free(struct ref_transaction *transaction)\n> \tfree(transaction);\n> }\n> \n> +static void add_update_obj(struct ref_transaction *transaction,\n> +\t\t\t struct ref_update *update)\n> +{\n> +\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n> +\ttransaction->updates[transaction->nr++] = update;\n> +}\n> +\n> static struct ref_update *add_update(struct ref_transaction *transaction,\n> \t\t\t\t const char *refname)\n> {\n> @@ -791,8 +803,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n> \tstruct ref_update *update = xcalloc(1, sizeof(*update) + len);\n> \n> \tmemcpy((char *)update->refname, refname, len); /* includes NUL */\n> -\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n> -\ttransaction->updates[transaction->nr++] = update;\n> +\tadd_update_obj(transaction, update);\n> \treturn update;\n> }\n> \n> @@ -1130,11 +1141,87 @@ int refs_init_db(struct strbuf *err, int shared)\n> \treturn the_refs_backend->init_db(err, shared);\n> }\n> \n> +/*\n> + * Special case for non-normal refs. For symbolic-refs when\n> + * REF_NODEREF is not turned on, we dereference them here and replace\n> + * updates to the symbolic refs with updates to the underlying ref.\n> + * Then we do our own reflogging for the symbolic ref.\n> + *\n> + * We move other non-normal ref updates with into a specially-created\n> + * files-backend transaction\n> + */\n\nExtra word? s/with//?\n\n> +static int move_abnormal_ref_updates(struct ref_transaction *transaction,\n> +\t\t\t\t struct ref_transaction *files_transaction,\n> +\t\t\t\t struct string_list *symrefs)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\tstruct ref_update *update = transaction->updates[i];\n> +\t\tconst char *resolved;\n> +\t\tint flags = 0;\n> +\t\tunsigned char sha1[20];\n> +\n> +\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n> +\t\t\tcontinue;\n> +\n> +\t\tresolved = resolve_ref_unsafe(update->refname, 0, sha1, &flags);\n> +\n> +\t\tif (update->flags & REF_NODEREF || !(flags & REF_ISSYMREF)) {\n> +\t\t\tint last;\n> +\n> +\t\t\tadd_update_obj(files_transaction, update);\n> +\t\t\t/*\n> +\t\t\t * Replace this transaction with the\n> +\t\t\t * last transaction, removing it from\n> +\t\t\t * the list of backend transactions\n> +\t\t\t */\n> +\t\t\tlast = --transaction->nr;\n> +\t\t\ttransaction->updates[i] = transaction->updates[last];\n\nThe \"last\" temporary variable could be trivially inlined.\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (resolved) {\n> +\t\t\tstruct ref_update *new_update;\n> +\t\t\tstruct string_list_item *item;\n> +\n> +\t\t\tif (ref_type(resolved) != REF_TYPE_NORMAL)\n> +\t\t\t\tdie(\"Non-normal symbolic ref `%s` points to non-normal ref `%s`\", update->refname, resolved);\n\nWe don't usually use backticks in error messages. Please use \"'\" instead.\n\nAlso, please store this error message into a \"strbuf *err\" and report it\nvia the usual mechanism.\n\n> +\t\t\tnew_update = xmalloc(sizeof(*new_update) +\n> +\t\t\t\t\t strlen(resolved) + 1);\n> +\t\t\tmemcpy(new_update, update, sizeof(*update));\n\nWouldn't it be preferable to replace this messy replacement code\n(including the memcpy(), which can't be checked by the type system) with\na call to ref_transaction_update() followed by moving the new update to\nthis position in the list and possibly tweaking some of its fields?\n\n> +\t\t\tif (update->flags & REF_HAVE_OLD &&\n> +\t\t\t hashcmp(sha1, update->old_sha1)) {\n> +\t\t\t\t/* consistency check failed */\n> +\t\t\t\tfree(new_update);\n> +\t\t\t\treturn -1;\n\nWe need an error message to be reported in this case; i.e., via a\n\"struct strbuf *err\" argument.\n\nBut actually, I don't understand why this check is needed here at all.\nIsn't it redundant with a similar check that will be done later (and\nproperly, under lock) as part of the main ref_transaction_commit()?\n\n> +\t\t\t} else {\n> +\t\t\t\thashcpy(update->old_sha1, sha1);\n> +\t\t\t}\n> +\n> +\t\t\tstrcpy((char *)new_update->refname, resolved);\n> +\t\t\ttransaction->updates[i] = new_update;\n> +\n> +\t\t\titem = string_list_append(symrefs, update->refname);\n> +\t\t\titem->util = new_update;\n> +\t\t\tfree(update);\n> +\t\t}\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\t\t struct strbuf *err)\n> {\n> \tint ret = -1;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list symrefs = STRING_LIST_INIT_DUP;\n> +\tstruct string_list_item *item;\n> +\tstruct ref_transaction *files_transaction = NULL;\n> \n> \tassert(err);\n> \n> @@ -1146,6 +1233,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \t\treturn 0;\n> \t}\n> \n> +\tif (the_refs_backend != &refs_be_files) {\n> +\t\tfiles_transaction = ref_transaction_begin(err);\n> +\t\tif (!files_transaction)\n> +\t\t\tdie(\"%s\", err->buf);\n\nI think dying here is too abrupt. Some callers try to recover from a\nfailed ref_transaction_commit(). Couldn't you \"goto done\" and let the\ncaller deal with err?\n\n> +\t\tret = move_abnormal_ref_updates(transaction, files_transaction,\n> +\t\t\t\t\t\t&symrefs);\n> +\t\tif (ret)\n> +\t\t\tgoto done;\n> +\n> +\t\t/* files backend commit */\n> +\t\tif (ref_update_reject_duplicates(files_transaction,\n> +\t\t\t\t\t\t &files_affected_refnames,\n> +\t\t\t\t\t\t err)) {\n> +\t\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\t\tgoto done;\n> +\t\t}\n\nIs it correct to reject_duplicates among \"abnormal\" references and\n\"normal\" references separately? ***\n\n> +\t}\n> +\n> +\t/* main backend commit */\n> \tif (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {\n> \t\tret = TRANSACTION_GENERIC_ERROR;\n> \t\tgoto done;\n> @@ -1153,8 +1260,35 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \n> \tret = the_refs_backend->transaction_commit(transaction,\n> \t\t\t\t\t\t &affected_refnames, err);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tif (the_refs_backend != &refs_be_files) {\n\nThis conditional would perhaps be more to the point if expressed as \"if\n(files_transaction)\".\n\n> +\t\tret = refs_be_files.transaction_commit(files_transaction,\n> +\t\t\t\t\t\t &files_affected_refnames,\n> +\t\t\t\t\t\t err);\n> +\t\tif (ret) {\n> +\t\t\twarning(split_transaction_fail_warning);\n> +\t\t\tgoto done;\n> +\t\t}\n> +\n> +\t\t/* reflogging for dereferenced symbolic refs */\n> +\t\tfor_each_string_list_item(item, &symrefs) {\n> +\t\t\tstruct ref_update *update = item->util;\n> +\t\t\tif (files_log_ref_write(item->string, update->old_sha1,\n> +\t\t\t\t\t\tupdate->new_sha1,\n> +\t\t\t\t\t\tupdate->msg, update->flags, err))\n> +\t\t\t\twarning(\"failed to log ref update for symref %s\",\n> +\t\t\t\t\titem->string);\n> +\t\t}\n\nI think this code is incorrect because it doesn't lock the symbolic\nreference before modifying its reflog (though I seem to recall that the\nold code was buggy in this respect, too).\n\nI wonder whether it would be simpler overall to leave the ref_update for\nthe symbolic ref in the files_transaction, but set a special internal\ninternal flag like REF_LOG_ONLY which tells the usual transaction_commit\ncode to add a reflog entry for update->old_sha1 to update->new_sha1,\nwithout actually changing the reference.\n\n> +\t}\n> +\n> done:\n> \tstring_list_clear(&affected_refnames, 0);\n> +\tstring_list_clear(&files_affected_refnames, 0);\n> +\tif (files_transaction)\n> +\t\tref_transaction_free(files_transaction);\n> +\tstring_list_clear(&symrefs, 0);\n> \treturn ret;\n> }\n> \n> @@ -1210,6 +1344,9 @@ int peel_ref(const char *refname, unsigned char *sha1)\n> int create_symref(const char *ref_target, const char *refs_heads_master,\n> \t\t const char *logmsg)\n> {\n> +\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n> +\t\treturn refs_be_files.create_symref(ref_target, refs_heads_master,\n> +\t\t\t\t\t\t logmsg);\n> \treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n> \t\t\t\t\t logmsg);\n> }\n> \n\nI very much like the idea of introducing special handling for symbolic\nreference updates within a transaction. In fact, I think I would go even\nfarther:\n\nLet's take the example of an update to HEAD, which currently points at\nrefs/heads/master. I think it would *always* be a good idea (i.e., even\nwhen only the files backend is in use) to split that ref_update into two\nref_updates:\n\n1. One to update refs/heads/master and add a reflog entry for that\n reference.\n\n2. One to add a reflog entry for HEAD (i.e. using the new REF_LOG_ONLY\n flag suggested above).\n\nWhy?\n\n* It ensures that both references are locked correctly while their\n reflogs are updated. (I believe the current code gets this wrong.)\n\n* It improves the reject_duplicates coverage, which (I think) currently\n wouldn't detect the conflict between a direct update of\n refs/heads/master with a simultaneous update of the same reference\n via HEAD.\n\n* It could later be generalized to an update that goes through multiple\n layers of symref indirection (though this would be a very low\n priority).\n\nThis might benefit the split-backend situation that you are implementing\nhere. You could first do the symref-splitting step I just described, and\n*then* separate the non-normal from the normal refs. I think the net\nresult would be simpler.\n\nThis patch is a lot to digest. I'm not yet confident that I have thought\nthrough all of the ramifications of this patch. I guess a few iterations\nwill be needed in any case.\n\nBy the way, all of the patches preceding this one that I haven't\ncommented on look OK to me.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274938","messageId":"567A6EE9.3030600@alum.mit.edu","threadId":"40926","inReplyTo":"20151205074444.GD21639@sigill.intra.peff.net","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T09:52:41Z","receivedAt":"2015-12-23T09:52:41Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/05/2015 08:44 AM, Jeff King wrote:\n> [...]\n> I think the config option needs to be extensions.refsBackendType, too,\n> per the logic in 00a09d5 (introduce \"extensions\" form of\n> core.repositoryformatversion, 2015-06-23). And I guess it needs to bump\n> core.repositoryformatversion to \"1\".\n\nI think also, strictly speaking, the extensions.refsBackendType option\nshould be ignored if core.repositoryFormatVersion is not \"1\". In\npractice, it probably makes more sense for the code to error out in that\ncase because it is likely the result of a bug.\n\nNote that if the user explicitly chooses the \"files\" backend, it would\nbe preferable to leave \"core.repositoryFormatVersion\" at \"0\" (assuming\nthat no other extension is being used) and leave\n\"extensions.refsBackendType\" unset. This approach creates a repository\nthat is compatible with older clients that don't know about\nrefsBackendTypes.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274941","messageId":"c21eb4a5d3a3a4886c45da0abe307fe1772e932e.1450869637.git.mhagger@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-14-git-send-email-dturner@twopensource.com","subject":"[PATCH] clone: use child_process for recursive checkouts","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T11:30:12Z","receivedAt":"2015-12-23T11:30:12Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Signed-off-by: Michael Haggerty \n---\nDavid, I think if you insert this patch before your\n\n 13/16 refs: allow ref backend to be set for clone\n\n, then the hunk in builtin/clone.c:checkout() of your patch becomes\ntrivial:\n\n\tif (refs_backend_type)\n\t\targv_array_pushf(&cmd.args, \"--refs-backend-type=%s\",\n\t\t\t\t refs_backend_type);\n\n builtin/clone.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex caae43e..53c2834 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -95,10 +95,6 @@ static struct option builtin_clone_options[] = {\n \tOPT_END()\n };\n \n-static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n-};\n-\n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n {\n \tstatic char *suffix[] = { \"/.git\", \"\", \".git/.git\", \".git\" };\n@@ -724,8 +720,14 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n-\t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\tif (!err && option_recursive) {\n+\t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\n+\t\tcmd.git_cmd = 1;\n+\t\targv_array_pushl(&cmd.args, \"submodule\", \"update\",\n+\t\t\t\t \"--init\", \"--recursive\", NULL);\n+\t\terr = run_command(&cmd);\n+\t}\n \n \treturn err;\n }\n-- \n2.6.4\n"},{"id":"274942","messageId":"567AA2DF.1020408@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-14-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T13:34:23Z","receivedAt":"2015-12-23T13:34:23Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> git init learns a new argument --refs-backend-type. Presently, only\n> \"files\" is supported, but later we will add other backends.\n> \n> When this argument is used, the repository's core.refsBackendType\n> configuration value is set, and the refs backend's initdb function is\n> used to set up the ref database.\n> \n> Signed-off-by: David Turner \n> ---\n> Documentation/git-init-db.txt | 2 +-\n> Documentation/git-init.txt | 6 +++++-\n> builtin/init-db.c | 10 ++++++++++\n> cache.h | 2 ++\n> config.c | 20 ++++++++++++++++++++\n> environment.c | 1 +\n> path.c | 32 ++++++++++++++++++++++++++++++--\n> refs.c | 8 ++++++++\n> refs.h | 12 ++++++++++++\n> setup.c | 10 ++++++++++\n> 10 files changed, 99 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt\n> index 648a6cd..72fbd71 100644\n> --- a/Documentation/git-init-db.txt\n> +++ b/Documentation/git-init-db.txt\n> @@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n> SYNOPSIS\n> --------\n> [verse]\n> -'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]]\n> +'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]] [--refs-backend-type=]\n> \n> \n> DESCRIPTION\n> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n> index 8174d27..9ea6753 100644\n> --- a/Documentation/git-init.txt\n> +++ b/Documentation/git-init.txt\n> @@ -12,7 +12,7 @@ SYNOPSIS\n> 'git init' [-q | --quiet] [--bare] [--template=]\n> \t [--separate-git-dir ]\n> \t [--shared[=]] [directory]\n> -\n> +\t [--refs-backend-type=]\n\nISTM that \"backend\" (used here in this option name, and in the manpage)\nis not such a meaningful term to users. Could we pick a less obscure\nterm? E.g., maybe \"--ref-storage=\"?\n\n\n> DESCRIPTION\n> -----------\n> @@ -113,6 +113,10 @@ does not exist, it will be created.\n> \n> --\n> \n> +--refs-backend-type=::\n> +Type of refs backend. Default is to use the original \"files\" backend,\n> +which stores ref data in files in .git/refs and .git/packed-refs.\n> +\n> TEMPLATE DIRECTORY\n> ------------------\n> \n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 4771e7e..44db591 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -204,6 +204,14 @@ static int create_default_files(const char *template_path)\n> \t\tadjust_shared_perm(get_git_dir());\n> \t}\n> \n> +\tif (refs_backend_type) {\n> +\t\tstruct refdb_config_data config_data = {NULL};\n> +\t\tgit_config_set(\"core.refsBackendType\", refs_backend_type);\n> +\t\tconfig_data.refs_backend_type = refs_backend_type;\n> +\t\tconfig_data.refs_base = get_git_dir();\n> +\t\tset_refs_backend(refs_backend_type, &config_data);\n> +\t}\n> +\n\nHere is the source of the \"void *data\" argument that puzzled me in patch\n08/16. I'm still puzzled. This code, which is later also used for the\nLMDB backend, *always* passes that function a \"struct refdb_config_data\n*\". So why is the argument declared to be \"void *\"?\n\nIf, on the other hand, you want to preserve a backend's freedom to\nrequire different extra data in this parameter, then this code in\ninit-db.c, and code like it elsewhere, would have to know about the\nreference backends so that it knows what data to pass to each one. In\nthat case, there would be no reason to make this function virtual; this\ncode could just as well call a different function (with a different\nsignature) depending on the backend that is in use.\n\nEither way, something seems strange here.\n\n(Remember that another alternative is to let the refs backend read\nwhatever specialized information it needs from the config by itself.)\n\n> \tif (refs_init_db(&err, shared_repository))\n> \t\tdie(\"failed to set up refs db: %s\", err.buf);\n> \n> @@ -469,6 +477,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n> \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n> \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n> \t\t\t N_(\"separate git dir from working tree\")),\n> +\t\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n> +\t\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n> \t\tOPT_END()\n> \t};\n> \n> diff --git a/cache.h b/cache.h\n> index 707455a..d1534db 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -696,6 +696,8 @@ extern enum object_creation_mode object_creation_mode;\n> \n> extern char *notes_ref_name;\n> \n> +extern const char *refs_backend_type;\n> +\n> extern int grafts_replace_parents;\n> \n> /*\n> diff --git a/config.c b/config.c\n> index 248a21a..210aa08 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -10,6 +10,7 @@\n> #include \"exec_cmd.h\"\n> #include \"strbuf.h\"\n> #include \"quote.h\"\n> +#include \"refs.h\"\n> #include \"hashmap.h\"\n> #include \"string-list.h\"\n> #include \"utf8.h\"\n> @@ -1207,6 +1208,25 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n> \t}\n> \n> \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n> +\t\tstruct refdb_config_data refdb_data = {NULL};\n> +\t\tchar *repo_config_copy;\n> +\n> +\t\t/*\n> +\t\t * make sure we always read the backend config from the\n> +\t\t * core section on startup\n> +\t\t */\n> +\t\tret += git_config_from_file(refdb_config, repo_config,\n> +\t\t\t\t\t &refdb_data);\n> +\n> +\t\trepo_config_copy = xstrdup(repo_config);\n> +\t\trefdb_data.refs_base = xstrdup(dirname(repo_config_copy));\n> +\t\tfree(repo_config_copy);\n> +\n> +\t\tif (refdb_data.refs_backend_type &&\n> +\t\t strcmp(refdb_data.refs_backend_type, \"files\")) {\n> +\t\t\tdie(\"Unexpected backend %s\", refdb_data.refs_backend_type);\n> +\t\t}\n\nThe refs code already maintains a list of valid backend names. It would\nbe better to ask it to validate this parameter than to maintain a second\nlist here. Or just call set_refs_backend() and let *it* fail.\n\n> +\n> \t\tret += git_config_from_file(fn, repo_config, data);\n> \t\tfound += 1;\n> \t}\n> diff --git a/environment.c b/environment.c\n> index 2da7fe2..8dbf0ab 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -66,6 +66,7 @@ int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> struct startup_info *startup_info;\n> unsigned long pack_size_limit_cfg;\n> +const char *refs_backend_type;\n> \n> #ifndef PROTECT_HFS_DEFAULT\n> #define PROTECT_HFS_DEFAULT 0\n> diff --git a/path.c b/path.c\n> index 3cd155e..86a8035 100644\n> --- a/path.c\n> +++ b/path.c\n> @@ -2,6 +2,7 @@\n> * Utilities for paths and pathnames\n> */\n> #include \"cache.h\"\n> +#include \"refs.h\"\n> #include \"strbuf.h\"\n> #include \"string-list.h\"\n> #include \"dir.h\"\n> @@ -510,9 +511,36 @@ int validate_headref(const char *path)\n> \tunsigned char sha1[20];\n> \tint fd;\n> \tssize_t len;\n> +\tstruct refdb_config_data refdb_data = {NULL, NULL};\n> +\n> +\tif (lstat(path, &st) < 0) {\n> +\t\tint backend_type_set;\n> +\t\tstruct strbuf config_path = STRBUF_INIT;\n> +\t\tif (path) {\n> +\t\t\tchar *pathdup = xstrdup(path);\n> +\t\t\tchar *git_dir = dirname(pathdup);\n> +\t\t\tstrbuf_addf(&config_path, \"%s/%s\", git_dir, \"config\");\n> +\t\t\tfree(pathdup);\n> +\t\t} else {\n> +\t\t\tstrbuf_addstr(&config_path, \"config\");\n> +\t\t}\n> \n> -\tif (lstat(path, &st) < 0)\n> -\t\treturn -1;\n> +\t\tif (git_config_from_file(refdb_config, config_path.buf, &refdb_data)) {\n> +\t\t\tstrbuf_release(&config_path);\n> +\t\t\treturn -1;\n> +\t\t}\n> +\n> +\t\tbackend_type_set = !!refdb_data.refs_backend_type;\n> +\t\tfree((void *)refdb_data.refs_backend_type);\n> +\t\tfree((void *)refdb_data.refs_base);\n> +\t\tstrbuf_release(&config_path);\n> +\t\t/*\n> +\t\t * Alternate backends are assumed to keep HEAD\n> +\t\t * in a valid state, so there's no need to do\n> +\t\t * further validation.\n> +\t\t */\n> +\t\treturn backend_type_set ? 0 : -1;\n> +\t}\n\nI'm not sure what problem you are trying to solve with this particular\nchange. Won't a lmdb-backed repository still have a \"HEAD\" file with the\nusual contents?\n\nIn any case, it seems like you are doing work that belongs at a higher\nlevel.\n\nvalidate_headref() has only one caller, is_git_directory(). The purpose\nof that function is to check whether a specified directory is a\nplausible Git directory, which it does by looking for some required files:\n\n* \"objects/\" (unless GIT_OBJECT_DIRECTORY is set)\n* \"refs/\"\n* \"HEAD\" (which must also have appropriate contents)\n\nNB: it doesn't care whether there is a \"config\" file. Presumably that\nwas invented later.\n\nIt seems to me that whether a repository should have a \"refs/\" and/or\n\"HEAD\", and if so what contents \"HEAD\" should have, or whether (for\nexample) it should have a \"refdb\" file, depends on what reference\nbackend is in use. But *that* depends on the contents of \"config\" or its\nabsence.\n\nYou are reading \"config\" down here in validate_headref(). It feels to me\nmore like is_git_directory() should read \"config\", determine which\nreference backend seems to be in use, and then ask *that* reference\nbackend whether the directory is a plausible Git directory. The files\nbackend would check for \"HEAD\" and \"refs/\" and the LMDB backend would\ncheck for \"HEAD\" and \"refdb/\".\n\nNote that this approach would require \"config\" to be read and parsed an\nextra time. That might be considered too expensive to do so early in the\nstartup code. I don't know.\n\nBy the way, this code ends up reading \"config\" files a bit more\npromiscuously than the old code. I don't think this raises any security\nconsiderations, but I wanted to point it out anyway. If somebody can put\na hostile \"config\" file in a directory where you are running git, then\nthey can probably put an \"objects/\", \"refs/\", and \"HEAD\" file there,\nwhich is all that is needed to get even the old code to read the\n\"config\" file.\n\n> \t/* Make sure it is a \"refs/..\" symlink */\n> \tif (S_ISLNK(st.st_mode)) {\n> diff --git a/refs.c b/refs.c\n> index e48e43a..96e1673 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -24,6 +24,14 @@ struct ref_be *the_refs_backend = &refs_be_files;\n> */\n> struct ref_be *refs_backends = &refs_be_files;\n> \n> +const char *refs_backend_type;\n> +\n> +void register_refs_backend(struct ref_be *be)\n> +{\n> +\tbe->next = refs_backends;\n> +\trefs_backends = be;\n> +}\n> +\n> /*\n> * This function is used to switch to an alternate backend.\n> */\n> diff --git a/refs.h b/refs.h\n> index c211b9e..c3670e8 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -510,6 +510,18 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n> \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n> \t\t\t void *policy_cb_data);\n> \n> +struct refdb_config_data {\n> +\tconst char *refs_backend_type;\n> +\tconst char *refs_base;\n> +};\n> +/*\n> + * Read the refdb configuration data out of the config file\n> + */\n> +int refdb_config(const char *var, const char *value, void *ptr);\n> +\n> +struct ref_be;\n> int set_refs_backend(const char *name, void *data);\n> \n> +void register_refs_backend(struct ref_be *be);\n> +\n> #endif /* REFS_H */\n> diff --git a/setup.c b/setup.c\n> index d343725..de6b8ac 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1,5 +1,6 @@\n> #include \"cache.h\"\n> #include \"dir.h\"\n> +#include \"refs.h\"\n> #include \"string-list.h\"\n> \n> static int inside_git_dir = -1;\n> @@ -263,6 +264,15 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n> \treturn ret;\n> }\n> \n> +int refdb_config(const char *var, const char *value, void *ptr)\n> +{\n> + struct refdb_config_data *cdata = ptr;\n> +\n> + if (!strcmp(var, \"core.refsbackendtype\"))\n> +\t cdata->refs_backend_type = xstrdup((char *)value);\n\nUnnecessary cast.\n\n> + return 0;\n> +}\n> +\n> /*\n> * Test if it looks like we're at a git directory.\n> * We want to see:\n\nI can guarantee that before long somebody will want a config option that\ncan be put in a global .gitconfig to choose the default refs backend\ntype for new repositories when it is not specified explicitly via the\ncommand line. But that can wait :-)\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274954","messageId":"567AA6DD.9000404@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-15-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 14/16] refs: allow ref backend to be set for clone","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T13:51:25Z","receivedAt":"2015-12-23T13:51:25Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> Add a new option, --refs-backend-type, to allow the ref backend type to\n> be set on new clones.\n> \n> Submodules must use the same ref backend as the parent repository, so\n\n^^^ is this verified anywhere? Hint: if not, it probably should be. For\nexample, imagine a future where an existing repository can be migrated\nin place from files -> lmdb. If somebody runs that command in a\nsubmodule, this invariant would be broken and *this* version of git\nwould have to at least detect the problem and refuse to corrupt things.\n\nI added CC to a couple submodules experts in case they have any\nfeedback. Are there any other code paths that can lead to submodules\nbeing newly cloned, where this option would also have to be passed?\n\n> we also pass the --refs-backend-type option option when cloning\n> submodules.\n> \n> Signed-off-by: David Turner \n> ---\n> Documentation/git-clone.txt | 4 ++++\n> builtin/clone.c | 27 +++++++++++++++++++++++++--\n> builtin/submodule--helper.c | 5 ++++-\n> cache.h | 1 +\n> 4 files changed, 34 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 6bf000d..431575b 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -14,6 +14,7 @@ SYNOPSIS\n> \t [-o ] [-b ] [-u ] [--reference ]\n> \t [--dissociate] [--separate-git-dir ]\n> \t [--depth ] [--[no-]single-branch]\n> +\t [--refs-backend-type=]\n> \t [--recursive | --recurse-submodules] [--] \n> \t []\n> \n> @@ -221,6 +222,9 @@ objects from the source repository into a pack in the cloned repository.\n> \tThe result is Git repository can be separated from working\n> \ttree.\n> \n> +--refs-backend-type=::\n> +\tType of refs backend. Default is to use the original files based\n> +\tbackend.\n> \n> ::\n> \tThe (possibly remote) repository to clone from. See the\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index caae43e..a53f341 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -92,11 +92,13 @@ static struct option builtin_clone_options[] = {\n> \t\t N_(\"separate git dir from working tree\")),\n> \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n> \t\t\tN_(\"set config inside the new repository\")),\n> +\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n> +\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n> \tOPT_END()\n> };\n> \n> static const char *argv_submodule[] = {\n> -\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n> +\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL, NULL\n> };\n> \n> static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n> @@ -724,8 +726,24 @@ static int checkout(void)\n> \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n> \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n> \n> -\tif (!err && option_recursive)\n> +\tif (!err && option_recursive) {\n> +\t\tconst char **backend_arg = argv_submodule;\n> +\t\tchar *new_backend_arg = NULL;\n> +\t\tif (refs_backend_type) {\n> +\t\t\twhile (*backend_arg)\n> +\t\t\t\t++backend_arg;\n> +\n> +\t\t\tnew_backend_arg = xmalloc(21 + strlen(refs_backend_type));\n> +\t\t\tsprintf(new_backend_arg, \"--refs-backend-type=%s\",\n> +\t\t\t\trefs_backend_type);\n> +\t\t\t*backend_arg = new_backend_arg;\n> +\t\t}\n> \t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n> +\t\tif (refs_backend_type) {\n> +\t\t\tfree(new_backend_arg);\n> +\t\t\t*backend_arg = NULL;\n> +\t\t}\n> +\t}\n\nI sent a separate email about this hunk:\n\n\nhttp://mid.gmane.org/c21eb4a5d3a3a4886c45da0abe307fe1772e932e.1450869637.git.mhagger@alum.mit.edu\n\n> \n> \treturn err;\n> }\n> @@ -744,6 +762,11 @@ static void write_config(struct string_list *config)\n> \t\t\t\t\t write_one_config, NULL) < 0)\n> \t\t\tdie(\"unable to write parameters to config file\");\n> \t}\n> +\n> +\tif (refs_backend_type &&\n> +\t write_one_config(\"core.refsBackendType\",\n> +\t\t\t refs_backend_type, NULL) < 0)\n> +\t\tdie(\"unable to write backend parameter to config file\");\n> }\n> \n> static void write_refspec_config(const char *src_ref_prefix,\n> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> index f4c3eff..5c9ca4e 100644\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -140,7 +140,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url\n> \t\targv_array_pushl(&cp.args, \"--reference\", reference, NULL);\n> \tif (gitdir && *gitdir)\n> \t\targv_array_pushl(&cp.args, \"--separate-git-dir\", gitdir, NULL);\n> -\n> +\tif (refs_backend_type && *refs_backend_type) {\n> +\t\targv_array_push(&cp.args, \"--refs-backend-type\");\n> +\t\targv_array_push(&cp.args, refs_backend_type);\n> +\t}\n\nThis could be shortened to\n\n argv_array_pushf(&cp.args, \"--refs-backend-type=%s\", refs_backend_type);\n\n> \targv_array_push(&cp.args, url);\n> \targv_array_push(&cp.args, path);\n> \n> diff --git a/cache.h b/cache.h\n> index d1534db..8f2ca55 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -695,6 +695,7 @@ enum object_creation_mode {\n> extern enum object_creation_mode object_creation_mode;\n> \n> extern char *notes_ref_name;\n> +extern const char *refs_backend_type;\n> \n> extern const char *refs_backend_type;\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274956","messageId":"567AB07E.1010803@alum.mit.edu","threadId":"40926","inReplyTo":"1449102921-7707-16-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-12-23T14:32:30Z","receivedAt":"2015-12-23T14:32:30Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 12/03/2015 01:35 AM, David Turner wrote:\n> Add a database backend for refs using LMDB. This backend runs git\n> for-each-ref about 30% faster than the files backend with fully-packed\n> refs on a repo with ~120k refs. It's also about 4x faster than using\n> fully-unpacked refs. In addition, and perhaps more importantly, it\n> avoids case-conflict issues on OS X.\n> \n> LMDB has a few features that make it suitable for usage in git:\n> \n> 1. It is relatively lightweight; it requires only one header file, and\n> the library code takes under 64k at runtime.\n> \n> 2. It is well-tested: it's been used in OpenLDAP for years.\n> \n> 3. It's very fast. LMDB's benchmarks show that it is among\n> the fastest key-value stores.\n> \n> 4. It has a relatively simple concurrency story; readers don't\n> block writers and writers don't block readers.\n> \n> Ronnie Sahlberg's original version of this patchset used tdb. The\n> major disadvantage of tdb is that tdb is hard to build on OS X. It's\n> also not in homebrew. So lmdb seemed simpler.\n> \n> To test this backend's correctness, I hacked test-lib.sh and\n> test-lib-functions.sh to run all tests under the refs backend. Dozens\n> of tests use manual ref/reflog reading/writing, or create submodules\n> without passing --refs-backend-type to git init. If those tests are\n> changed to use the update-ref machinery or test-refs-lmdb-backend (or,\n> in the case of packed-refs, corrupt refs, and dumb fetch tests, are\n> skipped), the only remaining failing tests are the git-new-workdir\n> tests and the gitweb tests.\n> \n> Signed-off-by: David Turner \n> ---\n> .gitignore | 1 +\n> Documentation/config.txt | 7 +\n> Documentation/git-clone.txt | 2 +-\n> Documentation/git-init.txt | 3 +-\n> Documentation/technical/refs-lmdb-backend.txt | 50 +\n> Documentation/technical/repository-version.txt | 5 +\n> Makefile | 12 +\n> builtin/init-db.c | 10 +-\n> config.c | 20 +-\n> configure.ac | 33 +\n> contrib/workdir/git-new-workdir | 3 +\n> refs.h | 1 +\n> refs/lmdb-backend.c | 2054 ++++++++++++++++++++++++\n> setup.c | 22 +-\n> test-refs-lmdb-backend.c | 68 +\n> 15 files changed, 2279 insertions(+), 12 deletions(-)\n> create mode 100644 Documentation/technical/refs-lmdb-backend.txt\n> create mode 100644 refs/lmdb-backend.c\n> create mode 100644 test-refs-lmdb-backend.c\n> \n> diff --git a/.gitignore b/.gitignore\n> index 1c2f832..87d45a2 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -199,6 +199,7 @@\n> /test-path-utils\n> /test-prio-queue\n> /test-read-cache\n> +/test-refs-lmdb-backend\n> /test-regex\n> /test-revision-walking\n> /test-run-command\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index f617886..5fb25ed 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -496,6 +496,13 @@ core.repositoryFormatVersion::\n> \tInternal variable identifying the repository format and layout\n> \tversion.\n> \n> +core.refsBackendType::\n> +\tType of refs backend. Default is to use the original files\n> +\tbased backend. Set to 'lmdb' to activate the lmdb database\n> +\tbackend. If you use the lmdb backend,\n> +\tcore.repositoryFormatVersion must be set to 1, and\n> +\textensions.refBackend must be set to 'lmdb'.\n\nThis phrasing makes it sound like I can go into an existing repository,\nchange this repository setting, and *presto* I will have a LMDB-backed\nrepository. I suggest rewording it more along the lines of \"this setting\nreflects the refs backend that is currently in use\".\n\nAlso please see my earlier question about whether users should see the\nterm \"backend\" or whether other terms would be easier to understand.\n\n> +\n> core.sharedRepository::\n> \tWhen 'group' (or 'true'), the repository is made shareable between\n> \tseveral users in a group (making sure all the files and objects are\n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 431575b..739c116 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -224,7 +224,7 @@ objects from the source repository into a pack in the cloned repository.\n> \n> --refs-backend-type=::\n> \tType of refs backend. Default is to use the original files based\n> -\tbackend.\n> +\tbackend. Set to \"lmdb\" to activate the lmdb database backend.\n> \n> ::\n> \tThe (possibly remote) repository to clone from. See the\n> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n> index 9ea6753..bbe253f 100644\n> --- a/Documentation/git-init.txt\n> +++ b/Documentation/git-init.txt\n> @@ -115,7 +115,8 @@ does not exist, it will be created.\n> \n> --refs-backend-type=::\n> Type of refs backend. Default is to use the original \"files\" backend,\n> -which stores ref data in files in .git/refs and .git/packed-refs.\n> +which stores ref data in files in .git/refs and .git/packed-refs. Set\n> +to \"lmdb\" to activate the lmdb database backend.\n> \n> TEMPLATE DIRECTORY\n> ------------------\n> diff --git a/Documentation/technical/refs-lmdb-backend.txt b/Documentation/technical/refs-lmdb-backend.txt\n> new file mode 100644\n> index 0000000..c497ffc\n> --- /dev/null\n> +++ b/Documentation/technical/refs-lmdb-backend.txt\n> @@ -0,0 +1,50 @@\n> +Notes on the LMDB refs backend\n> +==============================\n> +\n> +Design:\n> +------\n> +\n> +Refs and reflogs are stored in a lmdb database in .git/refdb. All\n> +keys and values are \\0-terminated.\n> +\n> +Keys for refs are the name of the ref (e.g. refs/heads/master).\n> +Values are the value of the ref, in hex\n> +(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n> +\n> +All per-worktree refs (refs/bisect/* and HEAD) are store using\n\ns/store/stored/\n\n> +the traditional files-based backend.\n> +\n> +Reflogs are stored as a series of database entries.\n> +\n> +For non-empty reflogs, there is one entry per logged ref\n> +update. The key format is logs/[refname]\\0[timestamp]. The timestamp\n> +is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n> +This means that reflog entries are chronologically ordered. Because\n> +LMDB is a btree database, we can efficiently iterate over these keys.\n\nIs there a guarantee that the reflog entries for all references updated\nin a single transaction have the same timestamp? Is there a guarantee\nthat updates that happened in *different* transactions have different\ntimestamps? These might be useful properties if they are easy to\nimplement, because then one could deduce the scope of transactions from\nthe reflog. But of course, they admittedly go beyond what the files\nbackend offers and so are not a requirement.\n\nIs the timestamp in ASCII base 10, binary format, or or something else?\n\n> +For an empty reflog, there is a \"header\" entry to show that a reflog\n> +exists. The header has the same format as an ordinary reflog, but with\n> +a timeztamp of all zeros and an empty value.\n> +\n> +Reflog values are in the same format as the original files-based\n> +reflog.\n\nI assume that means that each entry has the same contents as one line\nfrom a reflog file. Does the value include a trailing LF? (From above I\nguess it does have a trailing NUL.)\n\nIs the timestamp within the reflog entry guaranteed to agree with the\ntimestamp in the key? This might be a convenient property for debugging.\n\n> +Weaknesses:\n> +-----------\n> +\n> +The reflog format is somewhat inefficient: a binary format could store\n> +reflog date/time information in somewhat less space.\n> +\n> +The rsync and file:// transports don't work yet, because they\n> +don't use the refs API.\n> +\n> +git new-workdir is incompatible with the lmdb backend. Fortunately,\n> +git new-workdir is deprecated, and worktrees work fine.\n> +\n> +LMDB locks the entire database for write. Any other writer waits\n> +until the first writer is done before beginning. Readers do not wait\n> +for writers, and writers do not wait for readers. The underlying\n> +scheme is approximately MVCC; each reader's queries see the state of\n> +the database as-of the time that the reader acquired its read lock.\n> +This is not too far off from the files backend, which loads all refs\n> +into memory when one is requested.\n> diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\n> index 00ad379..04c085d 100644\n> --- a/Documentation/technical/repository-version.txt\n> +++ b/Documentation/technical/repository-version.txt\n> @@ -86,3 +86,8 @@ for testing format-1 compatibility.\n> When the config key `extensions.preciousObjects` is set to `true`,\n> objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n> `git repack -d`).\n> +\n> +`refBackend`\n> +~~~~~~~~~~~~\n> +This extension allows the user of alternate ref backends. The only\n> +defined value is `lmdb`.\n\ns/user/use/\n\n> diff --git a/Makefile b/Makefile\n> index 5bd68e0..77b96d9 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1037,6 +1037,17 @@ ifdef USE_LIBPCRE\n> \tEXTLIBS += -lpcre\n> endif\n> \n> +ifdef USE_LIBLMDB\n> +\tBASIC_CFLAGS += -DUSE_LIBLMDB\n> +\tifdef LIBLMDBDIR\n> +\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n> +\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n> +\tendif\n> +\tEXTLIBS += -llmdb\n> +\tLIB_OBJS += refs/lmdb-backend.o\n> +\tTEST_PROGRAMS_NEED_X += test-refs-lmdb-backend\n> +endif\n> +\n> ifdef HAVE_ALLOCA_H\n> \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n> endif\n> @@ -2124,6 +2135,7 @@ GIT-BUILD-OPTIONS: FORCE\n> \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n> \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n> \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n> +\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n> \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n> \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n> \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 44db591..1eb2feb 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -179,6 +179,7 @@ static int create_default_files(const char *template_path)\n> \tint reinit;\n> \tint filemode;\n> \tstruct strbuf err = STRBUF_INIT;\n> +\tint repo_version = 0;\n> \n> \t/* Just look for `init.templatedir` */\n> \tgit_config(git_init_db_config, NULL);\n> @@ -209,7 +210,14 @@ static int create_default_files(const char *template_path)\n> \t\tgit_config_set(\"core.refsBackendType\", refs_backend_type);\n> \t\tconfig_data.refs_backend_type = refs_backend_type;\n> \t\tconfig_data.refs_base = get_git_dir();\n> +#ifdef USE_LIBLMDB\n> +\t\tregister_refs_backend(&refs_be_lmdb);\n> +#endif\n> \t\tset_refs_backend(refs_backend_type, &config_data);\n> +\t\tif (!strcmp(refs_backend_type, \"lmdb\")) {\n\nWouldn't expressing this condition as\n\n if (strcmp(refs_backend_type, \"files\"))\n\nbetter reflect the reason that we might need to set\n\"extensions.refBackend\" and use repo_version \"1\"? (You might also need a\n\"refs_backend_type &&\" and/or a \"*refs_backend_type &&\" in there.)\n\n> +\t\t\tgit_config_set(\"extensions.refbackend\", \"lmdb\");\n> +\t\t\trepo_version = 1;\n> +\t\t}\n> \t}\n> \n> \tif (refs_init_db(&err, shared_repository))\n> @@ -229,7 +237,7 @@ static int create_default_files(const char *template_path)\n> \n> \t/* This forces creation of new config file */\n> \txsnprintf(repo_version_string, sizeof(repo_version_string),\n> -\t\t \"%d\", GIT_REPO_VERSION);\n> +\t\t \"%d\", repo_version);\n> \tgit_config_set(\"core.repositoryformatversion\", repo_version_string);\n> \n> \t/* Check filemode trustability */\n> diff --git a/config.c b/config.c\n> index 210aa08..779bb73 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1222,9 +1222,23 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n> \t\trefdb_data.refs_base = xstrdup(dirname(repo_config_copy));\n> \t\tfree(repo_config_copy);\n> \n> -\t\tif (refdb_data.refs_backend_type &&\n> -\t\t strcmp(refdb_data.refs_backend_type, \"files\")) {\n> -\t\t\tdie(\"Unexpected backend %s\", refdb_data.refs_backend_type);\n> +\t\tif (!refdb_data.refs_backend_type)\n> +\t\t\trefdb_data.refs_backend_type = \"\";\n> +\n> +\t\tif ((!*refdb_data.refs_backend_type) ||\n> +\t\t (!strcmp(refdb_data.refs_backend_type, \"files\"))) {\n> +\t\t\t/* default backend, nothing to do */\n> +\t\t} else if (!strcmp(refdb_data.refs_backend_type, \"lmdb\")) {\n> +\n> +#ifdef USE_LIBLMDB\n> +\t\t\trefs_backend_type = refdb_data.refs_backend_type;\n> +\t\t\tregister_refs_backend(&refs_be_lmdb);\n> +\t\t\tset_refs_backend(refs_backend_type, &refdb_data);\n> +#else\n> +\t\t\tdie(\"Git was not built with USE_LIBLMDB, so the db refs backend is not available\");\n> +#endif\n> +\t\t} else {\n> +\t\t\tdie(\"Unknown ref backend type '%s'\", refdb_data.refs_backend_type);\n> \t\t}\n> \n> \t\tret += git_config_from_file(fn, repo_config, data);\n> diff --git a/configure.ac b/configure.ac\n> index 89e2590..3853bec 100644\n> --- a/configure.ac\n> +++ b/configure.ac\n> @@ -271,6 +271,24 @@ AS_HELP_STRING([], [ARG can be also prefix for libpcre library and hea\n> dnl it yet.\n> \tGIT_CONF_SUBST([LIBPCREDIR])\n> fi)\n> +\n> +USE_LIBLMDB=YesPlease\n> +AC_ARG_WITH(liblmdb,\n> +AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])\n> +AS_HELP_STRING([], [ARG can be also prefix for liblmdb library and headers]),\n> + if test \"$withval\" = \"no\"; then\n> +\tUSE_LIBLMDB=\n> + elif test \"$withval\" = \"yes\"; then\n> +\tUSE_LIBLMDB=YesPlease\n> + else\n> +\tUSE_LIBLMDB=YesPlease\n> +\tLIBLMDBDIR=$withval\n> +\tAC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])\n> + dnl USE_LIBLMDB can still be modified below, so don't substitute\n> + dnl it yet.\n> +\tGIT_CONF_SUBST([LIBLMDBDIR])\n> + fi)\n> +\n> #\n> # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n> AC_FUNC_ALLOCA\n> @@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])\n> \n> fi\n> \n> +if test -n \"$USE_LIBLMDB\"; then\n> +\n> +GIT_STASH_FLAGS($LIBLMDBDIR)\n> +\n> +AC_CHECK_LIB([lmdb], [mdb_env_open],\n> +[USE_LIBLMDB=YesPlease],\n> +[USE_LIBLMDB=])\n> +\n> +GIT_UNSTASH_FLAGS($LIBLMDBDIR)\n> +\n> +GIT_CONF_SUBST([USE_LIBLMDB])\n> +\n> +fi\n> +\n> +\n> #\n> # Define NO_CURL if you do not have libcurl installed. git-http-pull and\n> # git-http-push are not built, and you cannot use http:// and https://\n> diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\n> index 888c34a..66b7ecf 100755\n> --- a/contrib/workdir/git-new-workdir\n> +++ b/contrib/workdir/git-new-workdir\n> @@ -28,6 +28,9 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n> git rev-parse --git-dir 2>/dev/null) ||\n> die \"Not a git repository: \\\"$orig_git\\\"\"\n> \n> +\n> +test \"$(git config core.refsbackendtype)\" = \"lmdb\" && die \"git-new-workdir is incompatible with the refs lmdb backend\"\n> +\n> case \"$git_dir\" in\n> .git)\n> \tgit_dir=\"$orig_git/.git\"\n> diff --git a/refs.h b/refs.h\n> index c3670e8..0cbfda9 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -520,6 +520,7 @@ struct refdb_config_data {\n> int refdb_config(const char *var, const char *value, void *ptr);\n> \n> struct ref_be;\n> +extern struct ref_be refs_be_lmdb;\n> int set_refs_backend(const char *name, void *data);\n> \n> void register_refs_backend(struct ref_be *be);\n> diff --git a/refs/lmdb-backend.c b/refs/lmdb-backend.c\n> [...]\n> [...]\n> [...]\n> [...]\n> [...]\n> diff --git a/setup.c b/setup.c\n> index de6b8ac..9724c0b 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -279,10 +279,11 @@ int refdb_config(const char *var, const char *value, void *ptr)\n> *\n> * - either an objects/ directory _or_ the proper\n> * GIT_OBJECT_DIRECTORY environment variable\n> - * - a refs/ directory\n> - * - either a HEAD symlink or a HEAD file that is formatted as\n> - * a proper \"ref:\", or a regular file HEAD that has a properly\n> - * formatted sha1 object name.\n> + * - a refdb/ directory or\n> + * - a refs/ directory\n> + * - either a HEAD symlink or a HEAD file that is formatted as\n> + * a proper \"ref:\", or a regular file HEAD that has a properly\n> + * formatted sha1 object name.\n\nThe indentation looks wrong here. I think the requirement is\n\n* - either a refdb/ directory or a refs/ directory\n* - either a HEAD symlink or a HEAD file that is formatted as\n* a proper \"ref:\", or a regular file HEAD that has a properly\n* formatted sha1 object name.\n\nthough the test of the *contents* of HEAD is currently skipped for\nnon-files backends.\n\n> */\n> int is_git_directory(const char *suspect)\n> {\n> @@ -313,8 +314,13 @@ int is_git_directory(const char *suspect)\n> \n> \tstrbuf_setlen(&path, len);\n> \tstrbuf_addstr(&path, \"/refs\");\n> -\tif (access(path.buf, X_OK))\n> -\t\tgoto done;\n> +\n> +\tif (access(path.buf, X_OK)) {\n> +\t\tstrbuf_setlen(&path, len);\n> +\t\tstrbuf_addstr(&path, \"/refdb\");\n> +\t\tif (access(path.buf, X_OK))\n> +\t\t\tgoto done;\n> +\t}\n> \n> \tret = 1;\n> done:\n> @@ -383,6 +389,10 @@ static int check_repo_format(const char *var, const char *value, void *cb)\n> \t\t\t;\n> \t\telse if (!strcmp(ext, \"preciousobjects\"))\n> \t\t\trepository_format_precious_objects = git_config_bool(var, value);\n> +#ifdef USE_LIBLMDB\n> +\t\telse if (!strcmp(ext, \"refbackend\") && !strcmp(value, \"lmdb\"))\n> +\t\t\t;\n> +#endif\n> \t\telse\n> \t\t\tstring_list_append(&unknown_extensions, ext);\n> \t}\n> diff --git a/test-refs-lmdb-backend.c b/test-refs-lmdb-backend.c\n> [...]\n\nI haven't actually reviewed the LMDB-specific code, just the glue code.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"274959","messageId":"20151223200133.GA19199@sigill.intra.peff.net","threadId":"40926","inReplyTo":"567A6EE9.3030600@alum.mit.edu","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-23T20:01:33Z","receivedAt":"2015-12-23T20:01:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 23, 2015 at 10:52:41AM +0100, Michael Haggerty wrote:\n\n> On 12/05/2015 08:44 AM, Jeff King wrote:\n> > [...]\n> > I think the config option needs to be extensions.refsBackendType, too,\n> > per the logic in 00a09d5 (introduce \"extensions\" form of\n> > core.repositoryformatversion, 2015-06-23). And I guess it needs to bump\n> > core.repositoryformatversion to \"1\".\n> \n> I think also, strictly speaking, the extensions.refsBackendType option\n> should be ignored if core.repositoryFormatVersion is not \"1\". In\n> practice, it probably makes more sense for the code to error out in that\n> case because it is likely the result of a bug.\n\nYeah, I agree. It's not wrong, but it may be a good indication the user\nis confused. Perhaps issuing a warning would be appropriate.\n\n> Note that if the user explicitly chooses the \"files\" backend, it would\n> be preferable to leave \"core.repositoryFormatVersion\" at \"0\" (assuming\n> that no other extension is being used) and leave\n> \"extensions.refsBackendType\" unset. This approach creates a repository\n> that is compatible with older clients that don't know about\n> refsBackendTypes.\n\nAlso agreed. This is going to be the case for basically every\n\"extension\" option. If the extension config key isn't present, there\nwill be some default backwards-compatible behavior, and for\ncompatibility, you are always better off turning off the extension than\nsetting turning it \"on\" with that default value. This is the case for\nthe \"preciousObjects\" extension: there is no point in setting it to\n\"false\" as opposed to removing it.\n\nSo I think the rules for setting an extension value should be:\n\n 1. If you're setting it to the default value, then remove it\n completely.\n\n 2. If you're removing it, and there are no other extensions, set\n repositoryFormatVersion back to 0.\n\nOf course there's currently no code that sets extension values. And\ndoing step 1 is going to be specific to each extension. So this a\ngeneral philosophy to follow, not something we would write in a\nfunction.\n\n-Peff\n"},{"id":"274960","messageId":"CAPig+cSaZRAXJ+sVZgFUbWcNMzqWGtH9rgqFXjupjtTJiQVYFw@mail.gmail.com","threadId":"40926","inReplyTo":"567AA6DD.9000404@alum.mit.edu","subject":"Re: [PATCH 14/16] refs: allow ref backend to be set for clone","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-12-23T20:23:25Z","receivedAt":"2015-12-23T20:23:25Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Dec 23, 2015 at 8:51 AM, Michael Haggerty wrote:\n> On 12/03/2015 01:35 AM, David Turner wrote:\n>> + if (refs_backend_type && *refs_backend_type) {\n>> + argv_array_push(&cp.args, \"--refs-backend-type\");\n>> + argv_array_push(&cp.args, refs_backend_type);\n>> + }\n>\n> This could be shortened to\n>\n> argv_array_pushf(&cp.args, \"--refs-backend-type=%s\", refs_backend_type);\n\nOr, without interpolation:\n\n argv_array_pushl(&cp.args, \"--refs-backend-type\", refs_backend_type, NULL);\n"},{"id":"275272","messageId":"20160103000623.GB14424@gmail.com","threadId":"40926","inReplyTo":"1449102921-7707-4-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"David Aguilar","fromEmail":"davvid@gmail.com","sentAt":"2016-01-03T00:06:23Z","receivedAt":"2016-01-03T00:06:23Z","isPatch":true,"sender":{"key":"davvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/13196?v=4"},"body":"Apologies for the late review, and this review should probably\ngo on patch 01 or 02 but I don't have it in my mbox atm...\n\nOn Wed, Dec 02, 2015 at 07:35:08PM -0500, David Turner wrote:\n> From: Ronnie Sahlberg \n> \n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n> refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n> refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n> 3 files changed, 112 insertions(+), 12 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 9562325..b9b0244 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1150,3 +1150,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n> {\n> \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n> }\n> +\n> +int head_ref(each_ref_fn fn, void *cb_data)\n> +{\n> +\treturn the_refs_backend->head_ref(fn, cb_data);\n> +}\n\nMy only comment is that it seems like having a single static global\nthe_refs_backend seems like it should be avoided.\n\nIt seems like the intention was to keep the existing interface\nas-is, which explains why this is using globals, but it seems\nlike the refs backend should be part of some \"application\ncontext\" struct on the stack or allocated during main(). It can\nthen be passed around in the API so that we do not need to have\na global.\n\nThat way the code will not be tied to a specific single\nthe_refs_backend and could in theory have multiple backends\nworking at the same time. If submodule were ever rewritten in C\nthen this could potentially be a useful feature.\n\nThat said, the refs backend is not the only global static data\nin git that would need to be factored out, but it wouldn't hurt\nto make this part a little more tidy.\n\nThoughts?\n-- \nDavid\n"},{"id":"275315","messageId":"xmqqziwlnp7d.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"20160103000623.GB14424@gmail.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-01-04T19:01:58Z","receivedAt":"2016-01-04T19:01:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Aguilar writes:\n\n>> diff --git a/refs.c b/refs.c\n>> index 9562325..b9b0244 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -1150,3 +1150,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n>> {\n>> \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n>> }\n>> +\n>> +int head_ref(each_ref_fn fn, void *cb_data)\n>> +{\n>> +\treturn the_refs_backend->head_ref(fn, cb_data);\n>> +}\n>\n> My only comment is that it seems like having a single static global\n> the_refs_backend seems like it should be avoided.\n>\n> It seems like the intention was to keep the existing interface\n> as-is, which explains why this is using globals, but it seems\n> like the refs backend should be part of some \"application\n> context\" struct on the stack or allocated during main(). It can\n> then be passed around in the API so that we do not need to have\n> a global.\n\nI think the ship is sailing in a different direction. The API to\ndeal with refs, possibly in multiple repositories, is that you use a\nsingle ref backend, and that backend is expected to handle refs in\nsubmodule repositories. I.e. refs.c::for_each_ref_in_submodule()\ncalls into the backend implementation of the same method. So from\nthat point of view, you cannot say \"the top level repository uses\nLMDB backend but this and that submodule refs are looked up by\nconsulting files backend\".\n\nIf we want to go the opposite direction, so that we can keep more\nthan one in-core representations of \"repository\" (what you called\n\"application context\") and optionally have different refs backend\nhandling different repositories, we most likely would want to\nrethink the part of the refs API that special cases the submodule\nrefs from within the current repository. Their implementation\nshould be excised from the API set, and instead made to be a set\nof thin wrapper functions that explicitly refer to a repository\ninstance that represents the submodule.\n\nThe refs and the object database go hand in hand (i.e. the former\ngives the anchoring points to keep objects in the latter alive), so\nan instance of \"repository\" would hold \"the_refs_backend\", \"objects[]\",\nand possibly \"the_index\" for that repository.\n\nThe caching of already read refs is a responsiblity of each ref\nbackend in the current codebase. I am not sure if that is a good\nplacement, or a single implementation of the caching layer should\ninstead sit on top of any and ll ref backends.\n"},{"id":"275316","messageId":"CAL=YDW=3JE==2s7Y3-fVRnTgaAzT674CAnypuXOKynpfD0q=Bg@mail.gmail.com","threadId":"40926","inReplyTo":"20160103000623.GB14424@gmail.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2016-01-04T19:12:19Z","receivedAt":"2016-01-04T19:12:19Z","isPatch":true,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Sat, Jan 2, 2016 at 4:06 PM, David Aguilar wrote:\n> Apologies for the late review, and this review should probably\n> go on patch 01 or 02 but I don't have it in my mbox atm...\n>\n> On Wed, Dec 02, 2015 at 07:35:08PM -0500, David Turner wrote:\n>> From: Ronnie Sahlberg \n>>\n>> Signed-off-by: Ronnie Sahlberg \n>> Signed-off-by: David Turner \n>> ---\n>> refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n>> refs/files-backend.c | 41 +++++++++++++++++++++++++++------------\n>> refs/refs-internal.h | 29 ++++++++++++++++++++++++++++\n>> 3 files changed, 112 insertions(+), 12 deletions(-)\n>>\n>> diff --git a/refs.c b/refs.c\n>> index 9562325..b9b0244 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -1150,3 +1150,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n>> {\n>> return the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n>> }\n>> +\n>> +int head_ref(each_ref_fn fn, void *cb_data)\n>> +{\n>> + return the_refs_backend->head_ref(fn, cb_data);\n>> +}\n>\n> My only comment is that it seems like having a single static global\n> the_refs_backend seems like it should be avoided.\n>\n> It seems like the intention was to keep the existing interface\n> as-is, which explains why this is using globals, but it seems\n> like the refs backend should be part of some \"application\n> context\" struct on the stack or allocated during main(). It can\n> then be passed around in the API so that we do not need to have\n> a global.\n\nNot commenting on whether this is the right direction or not. A global\nvariable holding a methods table might not be most aesthetic design,\nbut there are practicalities.\n\nHowever, that kind of change would change the function signatures for\nall public refs functions and probably most private refs functions as\nwell and will likely have massive conflicts with almost any other\npatch that touches the refs code.\n\nIf doing this API change is desired it is probably best to do that as\na separate patch later that ONLY does this signature change and\nnothing else to make review easier and to possibly make merge conflict\nchanges easier to manage.\n\n\n\n>\n> That way the code will not be tied to a specific single\n> the_refs_backend and could in theory have multiple backends\n> working at the same time. If submodule were ever rewritten in C\n> then this could potentially be a useful feature.\n>\n> That said, the refs backend is not the only global static data\n> in git that would need to be factored out, but it wouldn't hurt\n>ice, to make this part a little more tidy.\n>\n> Thoughts?\n> --\n> David\n"},{"id":"275321","messageId":"xmqqegdxnlba.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"CAL=YDW=3JE==2s7Y3-fVRnTgaAzT674CAnypuXOKynpfD0q=Bg@mail.gmail.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-01-04T20:26:01Z","receivedAt":"2016-01-04T20:26:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ronnie Sahlberg writes:\n\n> Not commenting on whether this is the right direction or not. A global\n> variable holding a methods table might not be most aesthetic design,\n> but there are practicalities.\n>\n> However, that kind of change would change the function signatures for\n> all public refs functions and probably most private refs functions as\n> well and will likely have massive conflicts with almost any other\n> patch that touches the refs code.\n\nI do not think \"the_backend\" that is the default is wrong. Most\ncallers want to interact with the single default backend that shows\nrefs from the single default repository (i.e. ours), and having a\nset of functions that interact with that single default instance of\nthe ref data without having to explicitly pass it as a parameter\nmakes sense.\n\nIt is just like how we handle the in-core index. Most callers want\nto interact with the_index, the single default one, and we have a\nset of \"cache\" functions written back in Linus's days to access it\nwithout having to pass &the_index as the parameter. Later in April\n2007 when I invented the support for multiple in-core index, we kept\nthe traditional \"works on the default instance\" API in place but\nmade them thin wrappers to the \"works on the given instance\", i.e. a\nset of \"index\" functions that was introduced at da94faf6 (Merge\nbranch 'jc/the-index', 2007-04-24). Even after that change, most\ncallers that work on the in-core index intract with \"the_index\", the\nsingle default instance, via the \"cache\" functions.\n\nAn API enhancement to allow us to handle refs in multiple\nrepositories separately would be a very welcome move (it would get\nrid of the hacky interface for_each_ref_in_submodule(), for one\nthing), but even after that happens, I'd expect we would have most\ncallers asking to interact with \"the_backend\", the single default\ninstance.\n"},{"id":"275338","messageId":"20160105011711.GA31667@sigill.intra.peff.net","threadId":"40926","inReplyTo":"xmqqegdxnlba.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-01-05T01:17:11Z","receivedAt":"2016-01-05T01:17:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 04, 2016 at 12:26:01PM -0800, Junio C Hamano wrote:\n\n> An API enhancement to allow us to handle refs in multiple\n> repositories separately would be a very welcome move (it would get\n> rid of the hacky interface for_each_ref_in_submodule(), for one\n> thing), but even after that happens, I'd expect we would have most\n> callers asking to interact with \"the_backend\", the single default\n> instance.\n\nI am just a bystander here, but I think I agree. I do not know how we\nwill handle migrations from one format to another, but if we can\ninstantiate two separate ref backends at once, it seems like we should\nbe able to just for_each_ref() in the \"old\" backend, and update_ref() in\nthe new one.\n\nI did not dig in the code, but I think that is similar to why we have\nthe multiple-index code. It is not about juggling a bunch of independent\ninstances, but about moving data between them (though not specifically\nfor migration in the case of the index).\n\nAs much as it would be nice to clean this up before moving to multiple\nbackends, though, I don't think we should make it a pre-requisite. This\nis a difficult topic as it is, and I'd rather see us make incremental\nimprovement (backends, then hopefully more flexible mixing of backends)\nthan get bogged down and get neither.\n\n-Peff\n"},{"id":"275342","messageId":"xmqq37ucln4g.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"20160105011711.GA31667@sigill.intra.peff.net","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-01-05T03:29:51Z","receivedAt":"2016-01-05T03:29:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> As much as it would be nice to clean this up before moving to multiple\n> backends, though, I don't think we should make it a pre-requisite. This\n> is a difficult topic as it is, and I'd rather see us make incremental\n> improvement (backends, then hopefully more flexible mixing of backends)\n> than get bogged down and get neither.\n\nOh, yes, if I sounded like this needs to be cleaned up before it can\ngo in, that wasn't what I meant. After all, the single in-core\nindex support was adequate for our use even before the multiple\nin-core index support was introduced, and we still use the API to\naccess the single default instance in most of the codepaths. I'd\nexpect that the ref API will evolve in the same way.\n"},{"id":"275375","messageId":"568BC88C.6000703@alum.mit.edu","threadId":"40926","inReplyTo":"xmqqziwlnp7d.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-01-05T13:43:40Z","receivedAt":"2016-01-05T13:43:40Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/04/2016 08:01 PM, Junio C Hamano wrote:\n> David Aguilar writes:\n> [...]\n>> My only comment is that it seems like having a single static global\n>> the_refs_backend seems like it should be avoided.\n>> [...]\n> \n> I think the ship is sailing in a different direction. The API to\n> deal with refs, possibly in multiple repositories, is that you use a\n> single ref backend, and that backend is expected to handle refs in\n> submodule repositories. I.e. refs.c::for_each_ref_in_submodule()\n> calls into the backend implementation of the same method. So from\n> that point of view, you cannot say \"the top level repository uses\n> LMDB backend but this and that submodule refs are looked up by\n> consulting files backend\".\n> \n> If we want to go the opposite direction, so that we can keep more\n> than one in-core representations of \"repository\" (what you called\n> \"application context\") and optionally have different refs backend\n> handling different repositories, we most likely would want to\n> rethink the part of the refs API that special cases the submodule\n> refs from within the current repository. Their implementation\n> should be excised from the API set, and instead made to be a set\n> of thin wrapper functions that explicitly refer to a repository\n> instance that represents the submodule.\n\nI definitely think that the \"one and only one refs backend per process\"\nmodel is not a good long-term approach. For example, submodules are\nrelatively independent but are nevertheless sometimes accessed from a\nsingle process. I don't think insisting that they all use the same\nreferences backend is tenable over the long term.\n\nBut I haven't insisted on this flexibility yet because I don't think it\nwill be much extra work in total to retrofit it later. What we mostly\nwant to avoid is the need to rewrite all call sites more than once. The\ncurrent patch series hasn't had to rewrite callers at all, so we still\nhave a rewrite in reserve :-)\n\nActually, maybe we will never have to rewrite all callers. I rather\nthink that we will retain one simplified API for dealing with \"*this*\nrepository's references\" and a second for dealing with other repos' refs.\n\nBy the way, I think the next step towards the ability to work with\nmultiple backends at the same time would be to store a pointer to the\nrefs backend inside the ref_cache instance (possibly renaming the\nlatter) and allowing them both to be looked up by submodule name as is\nalready done by get_ref_cache(). I think this change could also be done\nwithout rewriting callers.\n\n> [...]\n> The caching of already read refs is a responsiblity of each ref\n> backend in the current codebase. I am not sure if that is a good\n> placement, or a single implementation of the caching layer should\n> instead sit on top of any and ll ref backends.\n\nI still dream about having compoundable reference backends, in which\ncase, ultimately, a \"CacheReferenceStore\" instance would optionally wrap\non top of an arbitrary other \"ReferenceStore\" instance (so to speak) to\ngive it caching functionality.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"275384","messageId":"1452012129.3892.32.camel@twopensource.com","threadId":"40926","inReplyTo":"567A3EB6.9000405@alum.mit.edu","subject":"Re: [PATCH 11/16] refs: move duplicate check to common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-05T16:42:09Z","receivedAt":"2016-01-05T16:42:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 07:27 +0100, Michael Haggerty wrote:\n> > +static int ref_update_reject_duplicates(struct ref_transaction\n> > *transaction,\n> > +\t\t\t\t\tstruct string_list\n> > *refnames,\n> > +\t\t\t\t\tstruct strbuf *err)\n> \n> I like that you extract this code into a function. Though it feels\n> awkward to have a function called \"ref_update_reject_duplicates()\"\n> that\n> has a side effect of filling the names into a string list. I think it\n> would feel more natural to call the function get_affected_refnames(),\n> and treat the duplicate check as an extra bonus.\n\n\nRenamed, thanks.\n\n> > int ref_transaction_commit(struct ref_transaction *transaction,\n> > \t\t\t struct strbuf *err)\n> > {\n> > -\treturn the_refs_backend->transaction_commit(transaction,\n> > err);\n> > +\tint ret = -1;\n> > +\tstruct string_list affected_refnames =\n> > STRING_LIST_INIT_NODUP;\n> > +\n> > +\tassert(err);\n> > +\n> > +\tif (transaction->state != REF_TRANSACTION_OPEN)\n> > +\t\tdie(\"BUG: commit called for transaction that is\n> > not open\");\n> > +\n> > +\tif (!transaction->nr) {\n> > +\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\tif (ref_update_reject_duplicates(transaction,\n> > &affected_refnames, err)) {\n> > +\t\tret = TRANSACTION_GENERIC_ERROR;\n> > +\t\tgoto done;\n> > +\t}\n> > +\n> > +\tret = the_refs_backend->transaction_commit(transaction,\n> > +\t\t\t\t\t\t \n> > &affected_refnames, err);\n> > +done:\n> > +\tstring_list_clear(&affected_refnames, 0);\n> > +\treturn ret;\n> > }\n> \n> Here you are avoiding a small amount of code duplication by calling\n> ref_update_reject_duplicates() here rather than in the backend\n> -specific\n> code. But you are doing so at the cost of having to compute\n> affected_refnames here and pass it (redundantly) to the backend's\n> transaction_commit function. This increases the coupling between\n> these\n> functions, which in my opinion is worse than the small amount of code\n> duplication. But maybe it's just me.\n\nHm. I'm trying to follow the principle that the backends should do as\nlittle as possible (and that the common code should do as much as\npossible). This reduces the cognitive load when writing new backends -\n- a new backend author need not know that the list of ref updates must\nbe unique.\n\nWhat do others think? \n"},{"id":"275386","messageId":"1452014787.3892.40.camel@twopensource.com","threadId":"40926","inReplyTo":"567AA2DF.1020408@alum.mit.edu","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-05T17:26:27Z","receivedAt":"2016-01-05T17:26:27Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 14:34 +0100, Michael Haggerty wrote:\n> On 12/03/2015 01:35 AM, David Turner wrote:\n> > git init learns a new argument --refs-backend-type. Presently,\n> > only\n> > \"files\" is supported, but later we will add other backends.\n> > \n> > When this argument is used, the repository's core.refsBackendType\n> > configuration value is set, and the refs backend's initdb function\n> > is\n> > used to set up the ref database.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > Documentation/git-init-db.txt | 2 +-\n> > Documentation/git-init.txt | 6 +++++-\n> > builtin/init-db.c | 10 ++++++++++\n> > cache.h | 2 ++\n> > config.c | 20 ++++++++++++++++++++\n> > environment.c | 1 +\n> > path.c | 32 ++++++++++++++++++++++++++++++-\n> > -\n> > refs.c | 8 ++++++++\n> > refs.h | 12 ++++++++++++\n> > setup.c | 10 ++++++++++\n> > 10 files changed, 99 insertions(+), 4 deletions(-)\n> > \n> > diff --git a/Documentation/git-init-db.txt b/Documentation/git-init\n> > -db.txt\n> > index 648a6cd..72fbd71 100644\n> > --- a/Documentation/git-init-db.txt\n> > +++ b/Documentation/git-init-db.txt\n> > @@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n> > SYNOPSIS\n> > --------\n> > [verse]\n> > -'git init-db' [-q | --quiet] [--bare] [-\n> > -template=] [--separate-git-dir ] [-\n> > -shared[=]]\n> > +'git init-db' [-q | --quiet] [--bare] [-\n> > -template=] [--separate-git-dir ] [-\n> > -shared[=]] [--refs-backend-type=]\n> > \n> > \n> > DESCRIPTION\n> > diff --git a/Documentation/git-init.txt b/Documentation/git\n> > -init.txt\n> > index 8174d27..9ea6753 100644\n> > --- a/Documentation/git-init.txt\n> > +++ b/Documentation/git-init.txt\n> > @@ -12,7 +12,7 @@ SYNOPSIS\n> > 'git init' [-q | --quiet] [--bare] [-\n> > -template=]\n> > \t [--separate-git-dir ]\n> > \t [--shared[=]] [directory]\n> > -\n> > +\t [--refs-backend-type=]\n> \n> ISTM that \"backend\" (used here in this option name, and in the\n> manpage)\n> is not such a meaningful term to users. Could we pick a less obscure\n> term? E.g., maybe \"--ref-storage=\"?\n\nI'm working on the rest now, but wanted to comment on this first. I\nwent ahead and made this change, but I'm not sure I like it. In the\ngit codebase, the concept will continue to be called \"backend\"; there\nare already-accepted patches using that terminology. Having two\nseparate names for the same thing seems confusing to me. We have this\nproblem already with the index (\"cache\" or \"staging area\"), which is a\nlittle annoying when grepping the code. \n\n(For refs-backend-type, the name of the config variable is a third\nseparate thing, where we'll have to choose between the internal and the\nexternal name). \n\nSo I think maybe we should just go with refs-backend here. What do\nothers think?\n"},{"id":"275389","messageId":"xmqqk2nnkio4.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"1452014787.3892.40.camel@twopensource.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-01-05T18:03:39Z","receivedAt":"2016-01-05T18:03:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> I'm working on the rest now, but wanted to comment on this first. I\n> went ahead and made this change, but I'm not sure I like it. In the\n> git codebase, the concept will continue to be called \"backend\"; there\n> are already-accepted patches using that terminology. Having two\n> separate names for the same thing seems confusing to me.\n\nWe have the option to update whatever \"are already-accepted\" [*1*].\nThat would allow us to uniformly call it \"ref storage\", if we wanted\nto.\n\nIn any case, we shouldn't be using an unqualified \"backend\" (or\n\"storage\" for that matter); we should always say \"ref\", i.e. either\n\"ref backend\" or \"ref storage\", in the name.\n\nBetween \"backend\" and \"storage\", I am slightly in favor of the\nlatter, but I am not good at naming things so...\n\n\n[Footnote]\n\n*1* Output from\n\n $ git grep backend master --\n\nseems to show me only \n\n master:refs.c: * The backend-independent part of the reference module.\n\nand all others are other kinds of backends, e.g. \"merge backend\",\n\"http-backend\", etc. so that may not be too bad.\n"},{"id":"275390","messageId":"1452018264.3892.45.camel@twopensource.com","threadId":"40926","inReplyTo":"xmqqk2nnkio4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-05T18:24:24Z","receivedAt":"2016-01-05T18:24:24Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-01-05 at 10:03 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > I'm working on the rest now, but wanted to comment on this first. \n> > I\n> > went ahead and made this change, but I'm not sure I like it. In\n> > the\n> > git codebase, the concept will continue to be called \"backend\";\n> > there\n> > are already-accepted patches using that terminology. Having two\n> > separate names for the same thing seems confusing to me.\n> \n> We have the option to update whatever \"are already-accepted\" [*1*].\n> That would allow us to uniformly call it \"ref storage\", if we wanted\n> to.\n> \n> In any case, we shouldn't be using an unqualified \"backend\" (or\n> \"storage\" for that matter); we should always say \"ref\", i.e. either\n> \"ref backend\" or \"ref storage\", in the name.\n> \n> Between \"backend\" and \"storage\", I am slightly in favor of the\n> latter, but I am not good at naming things so...\n> \n> \n> [Footnote]\n> \n> *1* Output from\n> \n> $ git grep backend master --\n> \n> seems to show me only \n> \n> master:refs.c: * The backend-independent part of the reference\n> module.\n> \n> and all others are other kinds of backends, e.g. \"merge backend\",\n> \"http-backend\", etc. so that may not be too bad.\n\nThere's refs/files-backend.c in master.\n\nI guess the argument for \"backend\" is that it is a better description\nof the struct. That is, \"a storage\" sounds funny. Usually \"storage\"\nis a mass noun. I guess we could call them \"storage backends\" (with\n\"ref-storage\" in the UI), which would split the difference. I guess\nI'll go with that, and we can decide later whether to rename those\nfiles.\n"},{"id":"275391","messageId":"xmqqfuybkg79.fsf@gitster.mtv.corp.google.com","threadId":"40926","inReplyTo":"568BC88C.6000703@alum.mit.edu","subject":"Re: [PATCH 03/16] refs: add methods for the ref iterators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-01-05T18:56:58Z","receivedAt":"2016-01-05T18:56:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> Actually, maybe we will never have to rewrite all callers. I rather\n> think that we will retain one simplified API for dealing with \"*this*\n> repository's references\" and a second for dealing with other repos' refs.\n\nYeah, I think the similarity with the multiple in-core index support\nPeff brought up holds true here as well.\n\n>> [...]\n>> The caching of already read refs is a responsiblity of each ref\n>> backend in the current codebase. I am not sure if that is a good\n>> placement, or a single implementation of the caching layer should\n>> instead sit on top of any and ll ref backends.\n>\n> I still dream about having compoundable reference backends, in which\n> case, ultimately, a \"CacheReferenceStore\" instance would optionally wrap\n> on top of an arbitrary other \"ReferenceStore\" instance (so to speak) to\n> give it caching functionality.\n\nI am not sure if we need or want the fully stackable backends, but I\ncan see that the implementation of the API could be structured like\nso:\n\n (1) Users of API would interact solely with the in-core caching\n layer when creating, reading, modifying, enumerating and\n deleting refs and contents of their logs. The caching layer\n has a handle for each in-core representation of a \"repository\",\n and there is a single default one, i.e. our repository. Also\n there is a way to ask for the in-core represention of a\n \"repository\" for submodules.\n\n (2) An instance of an in-core \"repository\" caching layer knows what\n \"storage backend\" the repository uses and this is used to\n dispatch the requests to suitable backends. The caching layer\n would be responsible for maintaining the validity of in-core\n cache it keeps while it relays the requests.\n\n (3) The implementation of for_each_ref_in_submodule() family of\n functions may need to be restructured; having to have the\n backend method for them would force storage backends to be\n aware of \"other\" repositories. Other \"only in this repository\"\n methods David and Ronnie's topics abstracts out of the files\n backend would remain the same.\n\nwhich may be clean and flexible enough for our purpose.\n\nJust to reiterate to avoid misunderstanding, I do not think that\nkind of restructuring has to be a prerequisite for the current\neffort to allow multiple backends, though.\n\nThanks.\n"},{"id":"275406","messageId":"1452039207.3892.93.camel@twopensource.com","threadId":"40926","inReplyTo":"567A5516.9070209@alum.mit.edu","subject":"Re: [PATCH 12/16] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T00:13:27Z","receivedAt":"2016-01-06T00:13:27Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 09:02 +0100, Michael Haggerty wrote:\n> + * We move other non-normal ref updates with into a specially\n> > -created\n> > + * files-backend transaction\n> > + */\n> \n> Extra word? s/with//?\n\nFixed, thanks.\n\n> > +\t\t\tadd_update_obj(files_transaction, update);\n> > +\t\t\t/*\n> > +\t\t\t * Replace this transaction with the\n> > +\t\t\t * last transaction, removing it from\n> > +\t\t\t * the list of backend transactions\n> > +\t\t\t */\n> > +\t\t\tlast = --transaction->nr;\n> > +\t\t\ttransaction->updates[i] = transaction\n> > ->updates[last];\n> \n> The \"last\" temporary variable could be trivially inlined.\n\nYes, but then the line would be > 80 characters.\n\n> > +\t\t\tcontinue;\n> > +\t\t}\n> > +\n> > +\t\tif (resolved) {\n> > +\t\t\tstruct ref_update *new_update;\n> > +\t\t\tstruct string_list_item *item;\n> > +\n> > +\t\t\tif (ref_type(resolved) != REF_TYPE_NORMAL)\n> > +\t\t\t\tdie(\"Non-normal symbolic ref `%s`\n> > points to non-normal ref `%s`\", update->refname, resolved);\n> \n> We don't usually use backticks in error messages. Please use \"'\"\n> instead.\n\nFixed.\n\n> Also, please store this error message into a \"strbuf *err\" and report\n> it\n> via the usual mechanism.\n\nFixed.\n\n> > +\t\t\tnew_update = xmalloc(sizeof(*new_update) +\n> > +\t\t\t\t\t strlen(resolved) +\n> > 1);\n> > +\t\t\tmemcpy(new_update, update,sizeof(*update));\n> z\n> Wouldn't it be preferable to replace this messy replacement code\n> (including the memcpy(), which can't be checked by the type system)\n> with\n> a call to ref_transaction_update() followed by moving the new update\n> to\n> this position in the list and possibly tweaking some of its fields?\n>\n> > +\t\t\tif (update->flags & REF_HAVE_OLD &&\n> > +\t\t\t hashcmp(sha1, update->old_sha1)) {\n> > +\t\t\t\t/* consistency check failed */\n> > +\t\t\t\tfree(new_update);\n> > +\t\t\t\treturn -1;\n> \n> We need an error message to be reported in this case; i.e., via a\n> \"struct strbuf *err\" argument.\n>\n> But actually, I don't understand why this check is needed here at\n> all.\n> Isn't it redundant with a similar check that will be done later (and\n> properly, under lock) as part of the main ref_transaction_commit(\n\nLooks like probably yes.\n\n> > +\t\t\t} else {\n> > +\t\t\t\thashcpy(update->old_sha1, sha1);\n> > +\t\t\t}\n> > +\n> > +\t\t\tstrcpy((char *)new_update->refname,\n> > resolved);\n> > +\t\t\ttransaction->updates[i] = new_update;\n> > +\n> > +\t\t\titem = string_list_append(symrefs, update\n> > ->refname);\n> > +\t\t\titem->util = new_update;\n> > +\t\t\tfree(update);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > int ref_transaction_commit(struct ref_transaction *transaction,\n> > \t\t\t struct strbuf *err)\n> > {\n> > \tint ret = -1;\n> > \tstruct string_list affected_refnames =\n> > STRING_LIST_INIT_NODUP;\n> > +\tstruct string_list files_affected_refnames =\n> > STRING_LIST_INIT_NODUP;\n> > +\tstruct string_list symrefs = STRING_LIST_INIT_DUP;\n> > +\tstruct string_list_item *item;\n> > +\tstruct ref_transaction *files_transaction = NULL;\n> > \n> > \tassert(err);\n> > \n> > @@ -1146,6 +1233,26 @@ int ref_transaction_commit(struct\n> > ref_transaction *transaction,\n> > \t\treturn 0;\n> > \t}\n> > \n> > +\tif (the_refs_backend != &refs_be_files) {\n> > +\t\tfiles_transaction = ref_transaction_begin(err);\n> > +\t\tif (!files_transaction)\n> > +\t\t\tdie(\"%s\", err->buf);\n> \n> I think dying here is too abrupt. Some callers try to recover from a\n> failed ref_transaction_commit(). Couldn't you \"goto done\" and let the\n> caller deal with err?\n\nSure.\n\n> > +\t\tret = move_abnormal_ref_updates(transaction,\n> > files_transaction,\n> > +\t\t\t\t\t\t&symrefs);\n> > +\t\tif (ret)\n> > +\t\t\tgoto done;\n> > +\n> > +\t\t/* files backend commit */\n> > +\t\tif\n> > (ref_update_reject_duplicates(files_transaction,\n> > +\t\t\t\t\t\t &files_affected_r\n> > efnames,\n> > +\t\t\t\t\t\t err)) {\n> > +\t\t\tret = TRANSACTION_GENERIC_ERROR;\n> > +\t\t\tgoto done;\n> > +\t\t}\n> \n> Is it correct to reject_duplicates among \"abnormal\" references and\n> \"normal\" references separately? ***\n\nYes. No reference is both normal and abnormal. \n\n> > +\t}\n> > +\n> > +\t/* main backend commit */\n> > \tif (ref_update_reject_duplicates(transaction,\n> > &affected_refnames, err)) {\n> > \t\tret = TRANSACTION_GENERIC_ERROR;\n> > \t\tgoto done;\n> > @@ -1153,8 +1260,35 @@ int ref_transaction_commit(struct\n> > ref_transaction *transaction,\n> > \n> > \tret = the_refs_backend->transaction_commit(transaction,\n> > \t\t\t\t\t\t \n> > &affected_refnames, err);\n> > +\tif (ret)\n> > +\t\tgoto done;\n> > +\n> > +\tif (the_refs_backend != &refs_be_files) {\n> \n> This conditional would perhaps be more to the point if expressed as\n> \"if\n> (files_transaction)\".\n\nOK.\n\n> > +\t\tret =\n> > refs_be_files.transaction_commit(files_transaction,\n> > +\t\t\t\t\t\t \n> > &files_affected_refnames,\n> > +\t\t\t\t\t\t err);\n> > +\t\tif (ret) {\n> > +\t\t\twarning(split_transaction_fail_warning);\n> > +\t\t\tgoto done;\n> > +\t\t}\n> > +\n> > +\t\t/* reflogging for dereferenced symbolic refs */\n> > +\t\tfor_each_string_list_item(item, &symrefs) {\n> > +\t\t\tstruct ref_update *update = item->util;\n> > +\t\t\tif (files_log_ref_write(item->string,\n> > update->old_sha1,\n> > +\t\t\t\t\t\tupdate->new_sha1,\n> > +\t\t\t\t\t\tupdate->msg,\n> > update->flags, err))\n> > +\t\t\t\twarning(\"failed to log ref update\n> > for symref %s\",\n> > +\t\t\t\t\titem->string);\n> > +\t\t}\n> \n> I think this code is incorrect because it doesn't lock the symbolic\n> reference before modifying its reflog (though I seem to recall that\n> the\n> old code was buggy in this respect, too).\n\n> I wonder whether it would be simpler overall to leave the ref_update\n> for\n> the symbolic ref in the files_transaction, but set a special internal\n> internal flag like REF_LOG_ONLY which tells the usual\n> transaction_commit\n> code to add a reflog entry for update->old_sha1 to update->new_sha1,\n> without actually changing the reference.\n\nHm. This seems to work -- see below for more details.\n\n> > +\t}\n> > +\n> > done:\n> > \tstring_list_clear(&affected_refnames, 0);\n> > +\tstring_list_clear(&files_affected_refnames, 0);\n> > +\tif (files_transaction)\n> > +\t\tref_transaction_free(files_transaction);\n> > +\tstring_list_clear(&symrefs, 0);\n> > \treturn ret;\n> > }\n> > \n> > @@ -1210,6 +1344,9 @@ int peel_ref(const char *refname, unsigned\n> > char *sha1)\n> > int create_symref(const char *ref_target, const char\n> > *refs_heads_master,\n> > \t\t const char *logmsg)\n> > {\n> > +\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n> > +\t\treturn refs_be_files.create_symref(ref_target,\n> > refs_heads_master,\n> > +\t\t\t\t\t\t logmsg);\n> > \treturn the_refs_backend->create_symref(ref_target,\n> > refs_heads_master,\n> > \t\t\t\t\t logmsg);\n> > }\n> > \n> \n> I very much like the idea of introducing special handling for \n> symbolic reference updates within a transaction. In fact, I think I \n> would go even farther:\n> \n> Let's take the example of an update to HEAD, which currently points\n> at\n> refs/heads/master. I think it would *always* be a good idea (i.e., \n> even when only the files backend is in use) to split that ref_update \n> into two ref_updates:\n> \n> 1. One to update refs/heads/master and add a reflog entry for that\n> reference.\n> \n> 2. One to add a reflog entry for HEAD (i.e. using the new \n> REF_LOG_ONLY flag suggested above).\n> \n> Why?\n> \n> * It ensures that both references are locked correctly while their\n> reflogs are updated. (I believe the current code gets this wrong.)\n\nIt's true that both refs are locked while their reflogs are updated.\nHowever, the symbolic ref isn't locked while the update to its referent\nhappens, which is still bad. I guess we could fix that later, since I\ndon't think this would make the situation any worse.\n\nOne thing to note is that there's a bunch of dereferencing code in\nfiles-backend.c which we could eliminate by doing the split outside the\nbackend. I've made the basic version of this change, but haven't yet\nlooked at whether and how we can eliminate that code. Tomorrow (I\nhope), I'll do some more work on that.\n\n> * It improves the reject_duplicates coverage, which (I think)\n> currently\n> wouldn't detect the conflict between a direct update of\n> refs/heads/master with a simultaneous update of the same reference\n> via HEAD.\n\nAs it happens, the current code handles this case indirectly; the ref\nis locked by the first update, so the update through the second can't\ndo get the lock, causing the whole transaction to fail. After this\npatch, it is handled directly (since the main transaction now contains\nthe referent of HEAD).\n\n> * It could later be generalized to an update that goes through\n> multiple\n> layers of symref indirection (though this would be a very low\n> priority).\n> \n> This might benefit the split-backend situation that you are\n> implementing\n> here. You could first do the symref-splitting step I just described,\n> and\n> *then* separate the non-normal from the normal refs. I think the net\n> result would be simpler.\n> \n> This patch is a lot to digest. I'm not yet confident that I have\n> thought\n> through all of the ramifications of this patch. I guess a few\n> iterations\n> will be needed in any case.\n\nI'm sure that's true.\n\n> By the way, all of the patches preceding this one that I haven't\n> commented on look OK to me.\n\nThanks.\n"},{"id":"275430","messageId":"568D0267.7080707@alum.mit.edu","threadId":"40926","inReplyTo":"xmqqk2nnkio4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-01-06T12:02:47Z","receivedAt":"2016-01-06T12:02:47Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/05/2016 07:03 PM, Junio C Hamano wrote:\n> David Turner writes:\n> \n>> I'm working on the rest now, but wanted to comment on this first. I\n>> went ahead and made this change, but I'm not sure I like it. In the\n>> git codebase, the concept will continue to be called \"backend\"; there\n>> are already-accepted patches using that terminology. Having two\n>> separate names for the same thing seems confusing to me.\n> \n> We have the option to update whatever \"are already-accepted\" [*1*].\n> That would allow us to uniformly call it \"ref storage\", if we wanted\n> to.\n\n...whereas whatever we name the option, we have to live with forever\nbecause it is user-facing. It's more important to get the option name\ncorrect (though I agree that it would be nice for the nomenclature used\nin the code to be reminiscent of the option name).\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"275435","messageId":"CACsJy8DQMBfSh1re0jMk1W=d9gYa2=6jqO+Q6bqam5WBY3hfmg@mail.gmail.com","threadId":"40926","inReplyTo":"567AA2DF.1020408@alum.mit.edu","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-01-06T12:52:21Z","receivedAt":"2016-01-06T12:52:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Dec 23, 2015 at 8:34 PM, Michael Haggerty wrote:\n> On 12/03/2015 01:35 AM, David Turner wrote:\n>> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n>> index 8174d27..9ea6753 100644\n>> --- a/Documentation/git-init.txt\n>> +++ b/Documentation/git-init.txt\n>> @@ -12,7 +12,7 @@ SYNOPSIS\n>> 'git init' [-q | --quiet] [--bare] [--template=]\n>> [--separate-git-dir ]\n>> [--shared[=]] [directory]\n>> -\n>> + [--refs-backend-type=]\n>\n> ISTM that \"backend\" (used here in this option name, and in the manpage)\n> is not such a meaningful term to users. Could we pick a less obscure\n> term? E.g., maybe \"--ref-storage=\"?\n\n>From an (ex-)translator point of view, storage is also easier to\ntranslate than the technical term \"backend\". I know we do not\ntranslate option names, but whatever term you use usually show up in\nsome user-facing messages that need translating. But I do prefer\nbackend in source code, I think it expresses the idea much better than\nstorage.\n-- \nDuy\n"},{"id":"275454","messageId":"1452123697.3892.112.camel@twopensource.com","threadId":"40926","inReplyTo":"c21eb4a5d3a3a4886c45da0abe307fe1772e932e.1450869637.git.mhagger@alum.mit.edu","subject":"Re: [PATCH] clone: use child_process for recursive checkouts","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T23:41:37Z","receivedAt":"2016-01-06T23:41:37Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 12:30 +0100, Michael Haggerty wrote:\n> Signed-off-by: Michael Haggerty \n> ---\n> David, I think if you insert this patch before your\n> \n> 13/16 refs: allow ref backend to be set for clone\n> \n> , then the hunk in builtin/clone.c:checkout() of your patch becomes\n> trivial:\n> \n> \tif (refs_backend_type)\n> \t\targv_array_pushf(&cmd.args, \"--refs-backend-type=%s\",\n> \t\t\t\t refs_backend_type);\n\nInserted, thanks.\n"},{"id":"275455","messageId":"1452123719-22634-1-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"567A5516.9070209@alum.mit.edu","subject":"[PATCH/RFC v2 1/3] refs: allow log-only updates","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T23:41:57Z","receivedAt":"2016-01-06T23:41:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The refs infrastructure learns about log-only ref updates, which only\nupdate the reflog. Later, we will use this to separate symbolic\nreference resolution from ref updating.\n\nSigned-off-by: David Turner \n---\n refs/files-backend.c | 15 ++++++++++-----\n refs/refs-internal.h | 2 ++\n 2 files changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 176cf65..0800a57 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2821,7 +2821,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t}\n \t\t}\n \t}\n-\tif (commit_ref(lock)) {\n+\tif (!(flags & REF_LOG_ONLY) && commit_ref(lock)) {\n \t\terror(\"Couldn't set %s\", lock->ref_name);\n \t\tunlock_ref(lock);\n \t\treturn -1;\n@@ -3175,7 +3175,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\tgoto cleanup;\n \t\t}\n \t\tif ((update->flags & REF_HAVE_NEW) &&\n-\t\t !(update->flags & REF_DELETING)) {\n+\t\t !(update->flags & REF_DELETING) &&\n+\t\t !(update->flags & REF_LOG_ONLY)) {\n \t\t\tint overwriting_symref = ((update->type & REF_ISSYMREF) &&\n \t\t\t\t\t\t (update->flags & REF_NODEREF));\n \n@@ -3205,7 +3206,9 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->flags |= REF_NEEDS_COMMIT;\n \t\t\t}\n \t\t}\n-\t\tif (!(update->flags & REF_NEEDS_COMMIT)) {\n+\n+\t\tif (!(update->flags & REF_LOG_ONLY) &&\n+\t\t !(update->flags & REF_NEEDS_COMMIT)) {\n \t\t\t/*\n \t\t\t * We didn't have to write anything to the lockfile.\n \t\t\t * Close it to free up the file descriptor:\n@@ -3222,7 +3225,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \tfor (i = 0; i < n; i++) {\n \t\tstruct ref_update *update = updates[i];\n \n-\t\tif (update->flags & REF_NEEDS_COMMIT) {\n+\t\tif (update->flags & REF_NEEDS_COMMIT ||\n+\t\t update->flags & REF_LOG_ONLY) {\n \t\t\tif (commit_ref_update(update->backend_data,\n \t\t\t\t\t update->new_sha1, update->msg,\n \t\t\t\t\t update->flags, err)) {\n@@ -3242,7 +3246,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\tstruct ref_update *update = updates[i];\n \t\tstruct ref_lock *lock = update->backend_data;\n \n-\t\tif (update->flags & REF_DELETING) {\n+\t\tif (update->flags & REF_DELETING &&\n+\t\t !(update->flags & REF_LOG_ONLY)) {\n \t\t\tif (delete_ref_loose(lock, update->type, err)) {\n \t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n \t\t\t\tgoto cleanup;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 21ac680..649ba63 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -42,6 +42,8 @@\n * value to ref_update::flags\n */\n \n+#define REF_LOG_ONLY 0x80\n+\n /* Include broken references in a do_for_each_ref*() iteration */\n #define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n \n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"275456","messageId":"1452123719-22634-2-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1452123719-22634-1-git-send-email-dturner@twopensource.com","subject":"[PATCH/RFC v2 2/3] refs: resolve symbolic refs first","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T23:41:58Z","receivedAt":"2016-01-06T23:41:58Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Before committing ref updates, split symbolic ref updates into two\nparts: an update to the underlying ref, and a log-only update to the\nsymbolic ref. This ensures that both references are locked correctly\nwhile their reflogs are updated.\n\nIt is still possible to confuse git by concurrent updates, since the\nsplitting of symbolic refs does not happen under lock. So a symbolic ref\ncould be replaced by a plain ref in the middle of this operation, which\nwould lead to reflog discontinuities and missed old-ref checks.\n\nSigned-off-by: David Turner \n---\n refs.c | 70 +++++++++++++++++++++++++++++\n refs/files-backend.c | 123 +++++++++++++++++++++++++--------------------------\n refs/refs-internal.h | 8 ++++\n 3 files changed, 139 insertions(+), 62 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 38f29b3..2496b91 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1126,6 +1126,72 @@ int refs_init_db(struct strbuf *err, int shared)\n \treturn the_refs_backend->init_db(err, shared);\n }\n \n+/*\n+ * Special case for symbolic refs when REF_NODEREF is not turned on.\n+ * Dereference them here, mark them REF_LOG_ONLY, and add an update\n+ * for the underlying ref.\n+ */\n+static int dereference_symrefs(struct ref_transaction *transaction,\n+\t\t\t struct strbuf *err)\n+{\n+\tint i;\n+\tint nr = transaction->nr;\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tconst char *resolved;\n+\t\tunsigned char sha1[20];\n+\t\tint resolve_flags = 0;\n+\t\tint mustexist = (update->old_sha1 &&\n+\t\t\t\t !is_null_sha1(update->old_sha1));\n+\t\tint deleting = (update->flags & REF_HAVE_NEW) &&\n+\t\t\tis_null_sha1(update->new_sha1);\n+\t\tstruct ref_update *new_update;\n+\n+\t\tif (mustexist)\n+\t\t\tresolve_flags |= RESOLVE_REF_READING;\n+\t\tif (deleting)\n+\t\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME |\n+\t\t\t\tRESOLVE_REF_NO_RECURSE;\n+\n+\t\tif (strcmp(update->refname, \"HEAD\"))\n+\t\t\tupdate->flags |= REF_IS_NOT_HEAD;\n+\n+\t\tresolved = resolve_ref_unsafe(update->refname, resolve_flags,\n+\t\t\t\t\t sha1, &update->type);\n+\t\tif (!resolved) {\n+\t\t\t/*\n+\t\t\t * We'll try again to resolve this during\n+\t\t\t * commit and give a better error message\n+\t\t\t * then, but we know it's not a symbolic ref\n+\t\t\t * (or, indeed, any sort of ref).\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\thashcpy(update->read_sha1, sha1);\n+\n+\t\tif (update->flags & REF_NODEREF ||\n+\t\t !(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\t/* Create a new transaction for the underlying ref */\n+\t\tif (ref_transaction_update(transaction,\n+\t\t\t\t\t resolved,\n+\t\t\t\t\t update->new_sha1,\n+\t\t\t\t\t (update->flags & REF_HAVE_OLD) ?\n+\t\t\t\t\t update->old_sha1 : NULL,\n+\t\t\t\t\t update->flags & ~REF_IS_NOT_HEAD,\n+\t\t\t\t\t update->msg, err))\n+\t\t\treturn -1;\n+\n+\t\t/* Make the symbolic ref update log-only */\n+\t\tupdate->flags |= REF_LOG_ONLY | REF_NODEREF;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n@@ -1142,6 +1208,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \t\treturn 0;\n \t}\n \n+\tret = dereference_symrefs(transaction, err);\n+\tif (ret)\n+\t\tgoto done;\n+\n \tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto done;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0800a57..ce73bc4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -7,7 +7,6 @@\n \n struct ref_lock {\n \tchar *ref_name;\n-\tchar *orig_ref_name;\n \tstruct lock_file *lk;\n \tstruct object_id old_oid;\n };\n@@ -1839,7 +1838,6 @@ static void unlock_ref(struct ref_lock *lock)\n \tif (lock->lk)\n \t\trollback_lock_file(lock->lk);\n \tfree(lock->ref_name);\n-\tfree(lock->orig_ref_name);\n \tfree(lock);\n }\n \n@@ -1890,6 +1888,7 @@ static int remove_empty_directories(struct strbuf *path)\n */\n static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t\t const unsigned char *read_sha1,\n \t\t\t\t\t const struct string_list *extras,\n \t\t\t\t\t const struct string_list *skip,\n \t\t\t\t\t unsigned int flags, int *type_p,\n@@ -1897,13 +1896,13 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n {\n \tstruct strbuf ref_file = STRBUF_INIT;\n \tstruct strbuf orig_ref_file = STRBUF_INIT;\n-\tconst char *orig_refname = refname;\n \tstruct ref_lock *lock;\n \tint last_errno = 0;\n \tint type, lflags;\n \tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n-\tint resolve_flags = 0;\n+\tint resolve_flags = RESOLVE_REF_NO_RECURSE;\n \tint attempts_remaining = 3;\n+\tint resolved;\n \n \tassert(err);\n \n@@ -1911,67 +1910,68 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \n \tif (mustexist)\n \t\tresolve_flags |= RESOLVE_REF_READING;\n-\tif (flags & REF_DELETING) {\n-\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n-\t\tif (flags & REF_NODEREF)\n-\t\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n-\t}\n \n-\trefname = resolve_ref_unsafe(refname, resolve_flags,\n-\t\t\t\t lock->old_oid.hash, &type);\n-\tif (!refname && errno == EISDIR) {\n-\t\t/*\n-\t\t * we are trying to lock foo but we used to\n-\t\t * have foo/bar which now does not exist;\n-\t\t * it is normal for the empty directory 'foo'\n-\t\t * to remain.\n-\t\t */\n-\t\tstrbuf_git_path(&orig_ref_file, \"%s\", orig_refname);\n-\t\tif (remove_empty_directories(&orig_ref_file)) {\n+\tif (type_p && *type_p & REF_ISSYMREF) {\n+\t\thashcpy(lock->old_oid.hash, read_sha1);\n+\t} else {\n+\n+\t\tif (flags & REF_DELETING)\n+\t\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n+\n+\t\tresolved = !!resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t\tlock->old_oid.hash, &type);\n+\t\tif (!resolved && errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * we are trying to lock foo but we used to\n+\t\t\t * have foo/bar which now does not exist;\n+\t\t\t * it is normal for the empty directory 'foo'\n+\t\t\t * to remain.\n+\t\t\t */\n+\t\t\tstrbuf_git_path(&orig_ref_file, \"%s\", refname);\n+\t\t\tif (remove_empty_directories(&orig_ref_file)) {\n+\t\t\t\tlast_errno = errno;\n+\t\t\t\tif (!verify_refname_available_dir(refname, extras, skip,\n+\t\t\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n+\t\t\t\t\tstrbuf_addf(err, \"there are still refs under '%s'\",\n+\t\t\t\t\t\t refname);\n+\t\t\t\tgoto error_return;\n+\t\t\t}\n+\t\t\tresolved = !!resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t\t\tlock->old_oid.hash, &type);\n+\t\t}\n+\n+\t\tif (type_p)\n+\t\t\t*type_p = type;\n+\t\tif (!resolved) {\n \t\t\tlast_errno = errno;\n-\t\t\tif (!verify_refname_available_dir(orig_refname, extras, skip,\n+\t\t\tif (last_errno != ENOTDIR ||\n+\t\t\t !verify_refname_available_dir(refname, extras, skip,\n \t\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n-\t\t\t\tstrbuf_addf(err, \"there are still refs under '%s'\",\n-\t\t\t\t\t orig_refname);\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t \"unable to resolve reference %s: %s\",\n+\t\t\t\t\t refname, strerror(last_errno));\n+\n+\t\t\tgoto error_return;\n+\t\t}\n+\t\t/*\n+\t\t * If the ref did not exist and we are creating it, make sure\n+\t\t * there is no existing packed ref whose name begins with our\n+\t\t * refname, nor a packed ref whose name is a proper prefix of\n+\t\t * our refname.\n+\t\t */\n+\t\tif (is_null_oid(&lock->old_oid) &&\n+\t\t verify_refname_available_dir(refname, extras, skip,\n+\t\t\t\t\t\t get_packed_refs(&ref_cache), err)) {\n+\t\t\tlast_errno = ENOTDIR;\n \t\t\tgoto error_return;\n \t\t}\n-\t\trefname = resolve_ref_unsafe(orig_refname, resolve_flags,\n-\t\t\t\t\t lock->old_oid.hash, &type);\n-\t}\n-\tif (type_p)\n-\t *type_p = type;\n-\tif (!refname) {\n-\t\tlast_errno = errno;\n-\t\tif (last_errno != ENOTDIR ||\n-\t\t !verify_refname_available_dir(orig_refname, extras, skip,\n-\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n-\t\t\tstrbuf_addf(err, \"unable to resolve reference %s: %s\",\n-\t\t\t\t orig_refname, strerror(last_errno));\n-\n-\t\tgoto error_return;\n-\t}\n-\t/*\n-\t * If the ref did not exist and we are creating it, make sure\n-\t * there is no existing packed ref whose name begins with our\n-\t * refname, nor a packed ref whose name is a proper prefix of\n-\t * our refname.\n-\t */\n-\tif (is_null_oid(&lock->old_oid) &&\n-\t verify_refname_available_dir(refname, extras, skip,\n-\t\t\t\t\t get_packed_refs(&ref_cache), err)) {\n-\t\tlast_errno = ENOTDIR;\n-\t\tgoto error_return;\n \t}\n-\n \tlock->lk = xcalloc(1, sizeof(struct lock_file));\n \n \tlflags = 0;\n-\tif (flags & REF_NODEREF) {\n-\t\trefname = orig_refname;\n+\tif (flags & REF_NODEREF)\n \t\tlflags |= LOCK_NO_DEREF;\n-\t}\n \tlock->ref_name = xstrdup(refname);\n-\tlock->orig_ref_name = xstrdup(orig_refname);\n \tstrbuf_git_path(&ref_file, \"%s\", refname);\n \n retry:\n@@ -2537,7 +2537,7 @@ static int files_rename_ref(const char *oldrefname, const char *newrefname,\n \n \tlogmoved = log;\n \n-\tlock = lock_ref_sha1_basic(newrefname, NULL, NULL, NULL, 0, NULL, &err);\n+\tlock = lock_ref_sha1_basic(newrefname, NULL, NULL, NULL, NULL, 0, NULL, &err);\n \tif (!lock) {\n \t\terror(\"unable to rename '%s' to '%s': %s\", oldrefname, newrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -2555,7 +2555,7 @@ static int files_rename_ref(const char *oldrefname, const char *newrefname,\n \treturn 0;\n \n rollback:\n-\tlock = lock_ref_sha1_basic(oldrefname, NULL, NULL, NULL, 0, NULL, &err);\n+\tlock = lock_ref_sha1_basic(oldrefname, NULL, NULL, NULL, NULL, 0, NULL, &err);\n \tif (!lock) {\n \t\terror(\"unable to lock %s for rollback: %s\", oldrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -2783,9 +2783,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\t\t int flags, struct strbuf *err)\n {\n \tclear_loose_ref_cache(&ref_cache);\n-\tif (log_ref_write(lock->ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0 ||\n-\t (strcmp(lock->ref_name, lock->orig_ref_name) &&\n-\t log_ref_write(lock->orig_ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0)) {\n+\tif (log_ref_write(lock->ref_name, lock->old_oid.hash, sha1, logmsg, flags, err) < 0) {\n \t\tchar *old_msg = strbuf_detach(err, NULL);\n \t\tstrbuf_addf(err, \"Cannot update the ref '%s': %s\",\n \t\t\t lock->ref_name, old_msg);\n@@ -2793,7 +2791,7 @@ static int commit_ref_update(struct ref_lock *lock,\n \t\tunlock_ref(lock);\n \t\treturn -1;\n \t}\n-\tif (strcmp(lock->orig_ref_name, \"HEAD\") != 0) {\n+\tif (flags & REF_IS_NOT_HEAD) {\n \t\t/*\n \t\t * Special hack: If a branch is updated directly and HEAD\n \t\t * points to it (may happen on the remote side of a push\n@@ -3157,6 +3155,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\tupdate->refname,\n \t\t\t\t((update->flags & REF_HAVE_OLD) ?\n \t\t\t\t update->old_sha1 : NULL),\n+\t\t\t\tupdate->read_sha1,\n \t\t\t\taffected_refnames, NULL,\n \t\t\t\tupdate->flags,\n \t\t\t\t&update->type,\n@@ -3408,7 +3407,7 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \tstruct ref_lock *lock;\n \tchar *log_file;\n \tint status = 0;\n-\tint type;\n+\tint type = 0;\n \tstruct strbuf err = STRBUF_INIT;\n \n \tmemset(&cb, 0, sizeof(cb));\n@@ -3421,7 +3420,7 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \t * reference itself, plus we might need to update the\n \t * reference if --updateref was specified:\n \t */\n-\tlock = lock_ref_sha1_basic(refname, sha1, NULL, NULL, 0, &type, &err);\n+\tlock = lock_ref_sha1_basic(refname, sha1, NULL, NULL, NULL, 0, &type, &err);\n \tif (!lock) {\n \t\terror(\"cannot lock ref '%s': %s\", refname, err.buf);\n \t\tstrbuf_release(&err);\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 649ba63..971f450 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -44,6 +44,8 @@\n \n #define REF_LOG_ONLY 0x80\n \n+#define REF_IS_NOT_HEAD 0x100\n+\n /* Include broken references in a do_for_each_ref*() iteration */\n #define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n \n@@ -159,6 +161,12 @@ struct ref_update {\n \t */\n \tunsigned char old_sha1[20];\n \t/*\n+\t * During the symbolic ref split stage, we resolve refs.\n+\t * We'll re-resolve non-symbolic refs once they are locked,\n+\t * but we store this to avoid re-resolving symbolic refs.\n+\t */\n+\tunsigned char read_sha1[20];\n+\t/*\n \t * One or more of REF_HAVE_NEW, REF_HAVE_OLD, REF_NODEREF,\n \t * REF_DELETING, and REF_ISPRUNING:\n \t */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"275458","messageId":"1452123719-22634-3-git-send-email-dturner@twopensource.com","threadId":"40926","inReplyTo":"1452123719-22634-1-git-send-email-dturner@twopensource.com","subject":"[PATCH/RFC v2 3/3] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T23:41:59Z","receivedAt":"2016-01-06T23:41:59Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Always handle non-normal (per-worktree or pseudo) refs in the files\nbackend instead of alternate backends.\n\nSometimes a ref transaction will update both a per-worktree ref and a\nnormal ref. For instance, an ordinary commit might update\nrefs/heads/master and HEAD (or at least HEAD's reflog).\n\nUpdates to normal refs continue to go through the chosen backend.\n\nUpdates to non-normal refs are moved to a separate files backend\ntransaction.\n\nSigned-off-by: David Turner \n---\n refs.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n refs/refs-internal.h | 2 ++\n 2 files changed, 82 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 2496b91..67b0280 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -9,6 +9,11 @@\n #include \"object.h\"\n #include \"tag.h\"\n \n+static const char split_transaction_fail_warning[] =\n+\t\"A ref transaction was split across two refs backends. Part of the \"\n+\t\"transaction succeeded, but then the update to the per-worktree refs \"\n+\t\"failed. Your repository may be in an inconsistent state.\";\n+\n /*\n * We always have a files backend and it is the default.\n */\n@@ -781,6 +786,13 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \tfree(transaction);\n }\n \n+static void add_update_obj(struct ref_transaction *transaction,\n+\t\t\t struct ref_update *update)\n+{\n+\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n+\ttransaction->updates[transaction->nr++] = update;\n+}\n+\n static struct ref_update *add_update(struct ref_transaction *transaction,\n \t\t\t\t const char *refname)\n {\n@@ -788,8 +800,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \tstruct ref_update *update = xcalloc(1, sizeof(*update) + len);\n \n \tmemcpy((char *)update->refname, refname, len); /* includes NUL */\n-\tALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);\n-\ttransaction->updates[transaction->nr++] = update;\n+\tadd_update_obj(transaction, update);\n \treturn update;\n }\n \n@@ -1192,11 +1203,39 @@ static int dereference_symrefs(struct ref_transaction *transaction,\n \treturn 0;\n }\n \n+/*\n+ * Move all non-normal ref updates into a specially-created\n+ * files-backend transaction\n+ */\n+static int move_abnormal_ref_updates(struct ref_transaction *transaction,\n+\t\t\t\t struct ref_transaction *files_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tint last;\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\n+\t\tif (ref_type(update->refname) == REF_TYPE_NORMAL)\n+\t\t\tcontinue;\n+\n+\t\tlast = --transaction->nr;\n+\t\ttransaction->updates[i] = transaction->updates[last];\n+\t\tfiles_transaction->updates[files_transaction->nr++] = update;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t struct strbuf *err)\n {\n \tint ret = -1;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list_item *item;\n+\tstruct ref_transaction *files_transaction = NULL;\n \n \tassert(err);\n \n@@ -1212,6 +1251,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \tif (ret)\n \t\tgoto done;\n \n+\tif (the_refs_backend != &refs_be_files) {\n+\t\tfiles_transaction = ref_transaction_begin(err);\n+\t\tif (!files_transaction)\n+\t\t\tgoto done;\n+\n+\t\tret = move_abnormal_ref_updates(transaction, files_transaction,\n+\t\t\t\t\t\terr);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\t/* files backend commit */\n+\t\tif (get_affected_refnames(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err)) {\n+\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/* main backend commit */\n \tif (get_affected_refnames(transaction, &affected_refnames, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto done;\n@@ -1219,8 +1278,24 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \n \tret = the_refs_backend->transaction_commit(transaction,\n \t\t\t\t\t\t &affected_refnames, err);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (files_transaction) {\n+\t\tret = refs_be_files.transaction_commit(files_transaction,\n+\t\t\t\t\t\t &files_affected_refnames,\n+\t\t\t\t\t\t err);\n+\t\tif (ret) {\n+\t\t\twarning(split_transaction_fail_warning);\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n done:\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&files_affected_refnames, 0);\n+\tif (files_transaction)\n+\t\tref_transaction_free(files_transaction);\n \treturn ret;\n }\n \n@@ -1276,6 +1351,9 @@ int peel_ref(const char *refname, unsigned char *sha1)\n int create_symref(const char *ref_target, const char *refs_heads_master,\n \t\t const char *logmsg)\n {\n+\tif (ref_type(ref_target) != REF_TYPE_NORMAL)\n+\t\treturn refs_be_files.create_symref(ref_target, refs_heads_master,\n+\t\t\t\t\t\t logmsg);\n \treturn the_refs_backend->create_symref(ref_target, refs_heads_master,\n \t\t\t\t\t logmsg);\n }\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 971f450..dd035f5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -323,4 +323,6 @@ struct ref_storage_be {\n \tfor_each_replace_ref_fn *for_each_replace_ref;\n };\n \n+extern struct ref_storage_be refs_be_files;\n+\n #endif /* REFS_REFS_INTERNAL_H */\n-- \n2.4.2.749.g730654d-twtrsrc\n"},{"id":"275457","messageId":"1452123749.3892.114.camel@twopensource.com","threadId":"40926","inReplyTo":"567A5516.9070209@alum.mit.edu","subject":"Re: [PATCH 12/16] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-06T23:42:29Z","receivedAt":"2016-01-06T23:42:29Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 09:02 +0100, Michael Haggerty wrote:\n> I very much like the idea of introducing special handling for\n> symbolic\n> reference updates within a transaction. In fact, I think I would go\n> even\n> farther:\n> \n> Let's take the example of an update to HEAD, which currently points\n> at\n> refs/heads/master. I think it would *always* be a good idea (i.e.,\n> even\n> when only the files backend is in use) to split that ref_update into\n> two\n> ref_updates:\n> \n> 1. One to update refs/heads/master and add a reflog entry for that\n> reference.\n> \n> 2. One to add a reflog entry for HEAD (i.e. using the new\n> REF_LOG_ONLY\n> flag suggested above).\n\nHaving now implemented this, I think it's ugly.\n\nWhile committing a ref_update, we read the ref's old value. We need\nthis value for two things: 1. confirming that it matches old_sha1 in\nthe ref_update and 2. writing to the reflog. In the case of a symbolic\nref, we need to resolve it in order to get its old value. But we've\nalready resolved it during the splitting step (in order to know that it\nwas a symbolic ref). So we're duplicating effort. We can store the\nresolved SHA from the split phase (along with the resolved ref type),\nbut that's just leaking more weird details. \n\nIn addition, this doesn't actually solve the problem that this patch is\nintended to solve; I still have to split refs into normal and non\n-normal (files-backend). It's true that this splitting is simplified,\nsince we don't need to do weird things for the reflogs, but I'm not\nsure it's a net win.\n\nI'm going to send just these patches in reply to your mail (I hope),\nand see what you think before continuing down this path. \n"},{"id":"275462","messageId":"CAJo=hJt0U_XxYUmiGpZK+ZraA+CbcdgNMU-rnQfDS83Cve29ig@mail.gmail.com","threadId":"40926","inReplyTo":"CACsJy8DQMBfSh1re0jMk1W=d9gYa2=6jqO+Q6bqam5WBY3hfmg@mail.gmail.com","subject":"Re: [PATCH 13/16] init: allow alternate backends to be set for new repos","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-01-07T03:31:14Z","receivedAt":"2016-01-07T03:31:14Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Jan 6, 2016 at 4:52 AM, Duy Nguyen wrote:\n> On Wed, Dec 23, 2015 at 8:34 PM, Michael Haggerty wrote:\n>> On 12/03/2015 01:35 AM, David Turner wrote:\n>>> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n>>> index 8174d27..9ea6753 100644\n>>> --- a/Documentation/git-init.txt\n>>> +++ b/Documentation/git-init.txt\n>>> @@ -12,7 +12,7 @@ SYNOPSIS\n>>> 'git init' [-q | --quiet] [--bare] [--template=]\n>>> [--separate-git-dir ]\n>>> [--shared[=]] [directory]\n>>> -\n>>> + [--refs-backend-type=]\n>>\n>> ISTM that \"backend\" (used here in this option name, and in the manpage)\n>> is not such a meaningful term to users. Could we pick a less obscure\n>> term? E.g., maybe \"--ref-storage=\"?\n>\n> From an (ex-)translator point of view, storage is also easier to\n> translate than the technical term \"backend\". I know we do not\n> translate option names, but whatever term you use usually show up in\n> some user-facing messages that need translating. But I do prefer\n> backend in source code, I think it expresses the idea much better than\n> storage.\n\nAnd JGit calls these things RefDatabases. Because Java.\n\nAlso, its a table of ref -> SHA-1.\nAnd you put tables into databases.\nOr something.\n\n\n"},{"id":"275546","messageId":"1452257563.24133.5.camel@twopensource.com","threadId":"40926","inReplyTo":"1452123719-22634-3-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH/RFC v2 3/3] refs: always handle non-normal refs in files backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-08T12:52:43Z","receivedAt":"2016-01-08T12:52:43Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2016-01-06 at 18:41 -0500, David Turner wrote:\n+\t\tfiles_transaction->updates[files_transaction->nr++]\n> = update;\n\noops, need to allocate this (fixed on my branch).\n"},{"id":"275554","messageId":"1452269153.24133.14.camel@twopensource.com","threadId":"40926","inReplyTo":"567AB07E.1010803@alum.mit.edu","subject":"Re: [PATCH 15/16] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-01-08T16:05:53Z","receivedAt":"2016-01-08T16:05:53Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-12-23 at 15:32 +0100, Michael Haggerty wrote:\n> +core.refsBackendType::\n> > +\tType of refs backend. Default is to use the original files\n> > +\tbased backend. Set to 'lmdb' to activate the lmdb database\n> > +\tbackend. If you use the lmdb backend,\n> > +\tcore.repositoryFormatVersion must be set to 1, and\n> > +\textensions.refBackend must be set to 'lmdb'.\n> \n> This phrasing makes it sound like I can go into an existing\n> repository,\n> change this repository setting, and *presto* I will have a LMDB\n> -backed\n> repository. I suggest rewording it more along the lines of \"this\n> setting\n> reflects the refs backend that is currently in use\".\n> \n> Also please see my earlier question about whether users should see\n> the\n> term \"backend\" or whether other terms would be easier to understand.\n\nAdjusted, thanks.\n\n> > +All per-worktree refs (refs/bisect/* and HEAD) are store using\n> \n> s/store/stored/\n\nFixed.\n\n> > +the traditional files-based backend.\n> > +\n> > +Reflogs are stored as a series of database entries.\n> > +\n> > +For non-empty reflogs, there is one entry per logged ref\n> > +update. The key format is logs/[refname]\\0[timestamp]. The\n> > timestamp\n> > +is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n> > +This means that reflog entries are chronologically ordered. \n> > Because\n> > +LMDB is a btree database, we can efficiently iterate over these\n> > keys.\n> \n> Is there a guarantee that the reflog entries for all references\n> updated\n> in a single transaction have the same timestamp? Is there a guarantee\n> that updates that happened in *different* transactions have different\n> timestamps? These might be useful properties if they are easy to\n> implement, because then one could deduce the scope of transactions\n> from\n> the reflog. But of course, they admittedly go beyond what the files\n> backend offers and so are not a requirement.\n\nAll of this stuff is handled outside of the storage backend, so I don't\nwant to address it here. That is, this offers the same guarantees as\nthe files-based backend (neither of those, I think).\n\n> Is the timestamp in ASCII base 10, binary format, or or something\n> else?\n\nI've added the words \"network byte order\" to make it clear that this is\nbinary.\n\n> > +For an empty reflog, there is a \"header\" entry to show that a\n> > reflog\n> > +exists. The header has the same format as an ordinary reflog, but\n> > with\n> > +a timeztamp of all zeros and an empty value.\n> > +\n> > +Reflog values are in the same format as the original files-based\n> > +reflog.\n> \n> I assume that means that each entry has the same contents as one line\n> from a reflog file. Does the value include a trailing LF? (From above\n> I\n> guess it does have a trailing NUL.)\n\nTrailing LF, yes; will fix.\n\n> Is the timestamp within the reflog entry guaranteed to agree with the\n> timestamp in the key? This might be a convenient property for\n> debugging.\n\nYes; edited.\ndiff --git a/Makefile b/Makefile\n\n> > index 5bd68e0..77b96d9 100644\n> > --- a/Makefile\n> > +++ b/Makefile\n> > @@ -1037,6 +1037,17 @@ ifdef USE_LIBPCRE\n> > \tEXTLIBS += -lpcre\n> > endif\n> > \n> > +ifdef USE_LIBLMDB\n> > +\tBASIC_CFLAGS += -DUSE_LIBLMDB\n> > +\tifdef LIBLMDBDIR\n> > +\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n> > +\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib)\n> > $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n> > +\tendif\n> > +\tEXTLIBS += -llmdb\n> > +\tLIB_OBJS += refs/lmdb-backend.o\n> > +\tTEST_PROGRAMS_NEED_X += test-refs-lmdb-backend\n> > +endif\n> > +\n> > ifdef HAVE_ALLOCA_H\n> > \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n> > endif\n> > @@ -2124,6 +2135,7 @@ GIT-BUILD-OPTIONS: FORCE\n> > \t@echo NO_CURL=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(NO_CURL)))'\\' >>$@+\n> > \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(NO_EXPAT)))'\\' >>$@+\n> > \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n> > +\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n> > \t@echo NO_PERL=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(NO_PERL)))'\\' >>$@+\n> > \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(NO_PYTHON)))'\\' >>$@+\n> > \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst\n> > ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n> > diff --git a/builtin/init-db.c b/builtin/init-db.c\n> > index 44db591..1eb2feb 100644\n> > --- a/builtin/init-db.c\n> > +++ b/builtin/init-db.c\n> > @@ -179,6 +179,7 @@ static int create_default_files(const char\n> > *template_path)\n> > \tint reinit;\n> > \tint filemode;\n> > \tstruct strbuf err = STRBUF_INIT;\n> > +\tint repo_version = 0;\n> > \n> > \t/* Just look for `init.templatedir` */\n> > \tgit_config(git_init_db_config, NULL);\n> > @@ -209,7 +210,14 @@ static int create_default_files(const char\n> > *template_path)\n> > \t\tgit_config_set(\"core.refsBackendType\",\n> > refs_backend_type);\n> > \t\tconfig_data.refs_backend_type = refs_backend_type;\n> > \t\tconfig_data.refs_base = get_git_dir();\n> > +#ifdef USE_LIBLMDB\n> > +\t\tregister_refs_backend(&refs_be_lmdb);\n> > +#endif\n> > \t\tset_refs_backend(refs_backend_type, &config_data);\n> > +\t\tif (!strcmp(refs_backend_type, \"lmdb\")) {\n> \n> Wouldn't expressing this condition as\n> \n> if (strcmp(refs_backend_type, \"files\"))\n> \n> better reflect the reason that we might need to set\n> \"extensions.refBackend\" and use repo_version \"1\"?\n\nYeah.\n\n> (You might also need a\n> \"refs_backend_type &&\" and/or a \"*refs_backend_type &&\" in there.)\n\nI've just decided to set this var to be \"files\" by default, to simplify\na lot of this code.\n\n> > * - either an objects/ directory _or_ the proper\n> > * GIT_OBJECT_DIRECTORY environment variable\n> > - * - a refs/ directory\n> > - * - either a HEAD symlink or a HEAD file that is formatted as\n> > - * a proper \"ref:\", or a regular file HEAD that has a properly\n> > - * formatted sha1 object name.\n> > + * - a refdb/ directory or\n> > + * - a refs/ directory\n> > + * - either a HEAD symlink or a HEAD file that is formatted as\n> > + * a proper \"ref:\", or a regular file HEAD that has a\n> > properly\n> > + * formatted sha1 object name.\n> \n> The indentation looks wrong here. I think the requirement is\n> * - either a refdb/ directory or a refs/ directory\n> * - either a HEAD symlink or a HEAD file that is formatted as\n> * a proper \"ref:\", or a regular file HEAD that has a properly\n> * formatted sha1 object name.\n> \n> though the test of the *contents* of HEAD is currently skipped for\n> non-files backends.\n\nYes.\n\nThanks for the review. Hopefully another version by EOD, but we'll see.\n"}]}