{"thread":{"id":"40445","subject":"[PATCH v2 03/43] refs-be-files.c: rename refs to refs-be-files","startedAt":"2015-09-28T22:01:35Z","lastAt":"2015-10-12T09:00:08Z","messageCount":90,"participants":["David Turner","Junio C Hamano","Torsten Bögershausen","Dennis Kaarsemaker","Michael Haggerty","Jeff King","Michael Blume","Carlos Martín Nieto"],"isPatch":true,"patchVersion":2,"patchTotal":43},"messages":[{"id":"270845","messageId":"1443477738-32023-1-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":null,"subject":"[PATCH v2 00/43] lmdb ref backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:35Z","receivedAt":"2015-09-28T22:01:35Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This version rebases on top of pu. Mainly, this just moves a couple\nof methods to the backend.\n\nI added support for long symrefs to the lmdb backend. Peff added this\nto the files backend in pu, and there's a test for it.\n\nI cleaned up some formatting (long lines), and moved one patch to put\nit closer to its related patches. I added a comment to a bit I found confusing\nwhen re-reading the patch (in the lmdb backend patch).\n\nI re-ran the tests under lmdb, and everything that passed last time\npasses this time.\n\nDavid Turner (24):\n refs: make repack_without_refs and is_branch public\n refs-be-files.c: add method for for_each_reftype_...\n refs-be-files.c: add do_for_each_per_worktree_ref\n refs.c: move refname_is_safe to the common code\n refs.h: document make refname_is_safe and add it to header\n refs.c: move copy_msg to the common code\n refs.c: move peel_object to the common code\n refs.c: move should_autocreate_reflog to common code\n refs.c: add ref backend init function\n refs.c: add methods for reflog\n refs-be-files.c: add method to expire reflogs\n refs.c: add method for initial ref transaction commit\n initdb: move safe_create_dir into common code\n refs.c: add method for initializing refs db\n refs.c: make struct ref_transaction generic\n refs-be-files.c: add method to rename refs\n run-command: track total number of commands run\n refs: move some defines from refs-be-files.c to refs.h\n refs: make some files backend functions public\n refs: break out a ref conflict check\n refs: allow ref backend to be set for clone\n refs: add register_refs_backend\n refs: add LMDB refs backend\n refs: tests for db backend\n\nRonnie Sahlberg (19):\n refs.c: create a public version of verify_refname_available\n refs-be-files.c: rename refs to refs-be-files\n refs.c: add a new refs.c file to hold all common refs code\n refs.c: move update_ref to refs.c\n refs.c: move delete_ref and delete_refs to the common code\n refs.c: move read_ref_at to the common refs file\n refs.c: move the hidden refs functions to the common code\n refs.c: move dwim and friend functions to the common refs code\n refs.c: move warn_if_dangling_symref* to the common code\n refs.c: move read_ref, read_ref_full and ref_exists to the common code\n refs.c: move resolve_refdup to common\n refs.c: move check_refname_format to the common code\n refs.c: move is_branch to the common code\n refs.c: move prettify_refname to the common code\n refs.c: move ref iterators to the common code\n refs.c: move head_ref_namespaced to the common code\n refs-be-files.c: add a backend method structure with transaction\n functions\n refs-be-files.c: add methods for misc ref operations\n refs-be-files.c: add methods for the ref iterators\n\n .gitignore | 1 +\n Documentation/git-clone.txt | 4 +\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 6 +\n Documentation/technical/refs-be-lmdb.txt | 39 +\n Makefile | 13 +\n builtin/clone.c | 27 +-\n builtin/init-db.c | 39 +-\n builtin/submodule--helper.c | 5 +-\n cache.h | 13 +\n config.c | 27 +\n configure.ac | 33 +\n contrib/workdir/git-new-workdir | 2 +\n environment.c | 1 +\n path.c | 12 +\n refs-be-files.c | 3757 +++++++++++++++++++++\n refs-be-lmdb.c | 1984 +++++++++++\n refs.c | 5329 ++++++------------------------\n refs.h | 282 +-\n run-command.c | 2 +\n run-command.h | 1 +\n setup.c | 28 +-\n t/t1460-refs-be-db.sh | 1103 +++++++\n t/t1470-refs-be-db-reflog.sh | 353 ++\n test-refs-be-lmdb.c | 68 +\n 25 files changed, 8716 insertions(+), 4415 deletions(-)\n create mode 100644 Documentation/technical/refs-be-lmdb.txt\n create mode 100644 refs-be-files.c\n create mode 100644 refs-be-lmdb.c\n create mode 100755 t/t1460-refs-be-db.sh\n create mode 100755 t/t1470-refs-be-db-reflog.sh\n create mode 100644 test-refs-be-lmdb.c\n\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270886","messageId":"1443477738-32023-2-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:36Z","receivedAt":"2015-09-28T22:01:36Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nCreate a public version of verify_refname_available that backends can\nprovide.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 51 +++++++++++++++++++++++++++++----------------------\n refs.h | 17 +++++++++++++++++\n 2 files changed, 46 insertions(+), 22 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 132eff5..ce551e9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -279,7 +279,7 @@ struct ref_dir {\n * presence of an empty subdirectory does not block the creation of a\n * similarly-named reference. (The fact that reference names with the\n * same leading components can conflict *with each other* is a\n- * separate issue that is regulated by verify_refname_available().)\n+ * separate issue that is regulated by verify_refname_available_dir().)\n *\n * Please note that the name field contains the fully-qualified\n * reference (or subdirectory) name. Space could be saved by only\n@@ -911,11 +911,11 @@ static int nonmatching_ref_fn(struct ref_entry *entry, void *vdata)\n *\n * extras and skip must be sorted.\n */\n-static int verify_refname_available(const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t struct ref_dir *dir,\n-\t\t\t\t struct strbuf *err)\n+static int verify_refname_available_dir(const char *refname,\n+\t\t\t\t\tconst struct string_list *extras,\n+\t\t\t\t\tconst struct string_list *skip,\n+\t\t\t\t\tstruct ref_dir *dir,\n+\t\t\t\t\tstruct strbuf *err)\n {\n \tconst char *slash;\n \tint pos;\n@@ -2464,9 +2464,12 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t\t */\n \t\tstrbuf_git_path(&orig_ref_file, \"%s\", orig_refname);\n \t\tif (remove_empty_directories(&orig_ref_file)) {\n+\t\t\tstruct ref_dir *loose_refs;\n+\t\t\tloose_refs = get_loose_refs(&ref_cache);\n \t\t\tlast_errno = errno;\n-\t\t\tif (!verify_refname_available(orig_refname, extras, skip,\n-\t\t\t\t\t\t get_loose_refs(&ref_cache), err))\n+\t\t\tif (!verify_refname_available_dir(orig_refname, extras,\n+\t\t\t\t\t\t\t skip, loose_refs,\n+\t\t\t\t\t\t\t err))\n \t\t\t\tstrbuf_addf(err, \"there are still refs under '%s'\",\n \t\t\t\t\t orig_refname);\n \t\t\tgoto error_return;\n@@ -2479,8 +2482,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \tif (!refname) {\n \t\tlast_errno = errno;\n \t\tif (last_errno != ENOTDIR ||\n-\t\t !verify_refname_available(orig_refname, extras, skip,\n-\t\t\t\t\t get_loose_refs(&ref_cache), err))\n+\t\t !verify_refname_available_dir(orig_refname, extras, skip,\n+\t\t\t\t\t\t get_loose_refs(&ref_cache),\n+\t\t\t\t\t\t err))\n \t\t\tstrbuf_addf(err, \"unable to resolve reference %s: %s\",\n \t\t\t\t orig_refname, strerror(last_errno));\n \n@@ -2493,8 +2497,8 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,\n \t * our refname.\n \t */\n \tif (is_null_oid(&lock->old_oid) &&\n-\t verify_refname_available(refname, extras, skip,\n-\t\t\t\t get_packed_refs(&ref_cache), err)) {\n+\t verify_refname_available_dir(refname, extras, skip,\n+\t\t\t\t\t get_packed_refs(&ref_cache), err)) {\n \t\tlast_errno = ENOTDIR;\n \t\tgoto error_return;\n \t}\n@@ -3127,10 +3131,7 @@ static int rename_ref_available(const char *oldname, const char *newname)\n \tint ret;\n \n \tstring_list_insert(&skip, oldname);\n-\tret = !verify_refname_available(newname, NULL, &skip,\n-\t\t\t\t\tget_packed_refs(&ref_cache), &err)\n-\t\t&& !verify_refname_available(newname, NULL, &skip,\n-\t\t\t\t\t get_loose_refs(&ref_cache), &err);\n+\tret = !verify_refname_available(newname, NULL, &skip, &err);\n \tif (!ret)\n \t\terror(\"%s\", err.buf);\n \n@@ -3299,6 +3300,17 @@ static int should_autocreate_reflog(const char *refname)\n \t\t!strcmp(refname, \"HEAD\");\n }\n \n+int verify_refname_available(const char *newname, struct string_list *extra,\n+\t\t\t struct string_list *skip, struct strbuf *err)\n+{\n+\tstruct ref_dir *packed_refs = get_packed_refs(&ref_cache);\n+\tstruct ref_dir *loose_refs = get_loose_refs(&ref_cache);\n+\treturn verify_refname_available_dir(newname, extra, skip,\n+\t\t\t\t\t packed_refs, err) ||\n+\t\tverify_refname_available_dir(newname, extra, skip,\n+\t\t\t\t\t loose_refs, err);\n+}\n+\n /*\n * Create a reflog for a ref. If force_create = 0, the reflog will\n * only be created for certain refs (those for which\n@@ -4334,8 +4346,6 @@ static int ref_present(const char *refname,\n int initial_ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n {\n-\tstruct ref_dir *loose_refs = get_loose_refs(&ref_cache);\n-\tstruct ref_dir *packed_refs = get_packed_refs(&ref_cache);\n \tint ret = 0, i;\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n@@ -4378,10 +4388,7 @@ int initial_ref_transaction_commit(struct ref_transaction *transaction,\n \t\t\tdie(\"BUG: initial ref transaction with old_sha1 set\");\n \t\tif (verify_refname_available(update->refname,\n \t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t loose_refs, err) ||\n-\t\t verify_refname_available(update->refname,\n-\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t packed_refs, err)) {\n+\t\t\t\t\t err)) {\n \t\t\tret = TRANSACTION_NAME_CONFLICT;\n \t\t\tgoto cleanup;\n \t\t}\ndiff --git a/refs.h b/refs.h\nindex 6d30c98..79ea220 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -218,6 +218,23 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st\n int pack_refs(unsigned int flags);\n \n /*\n+ * Return true iff a reference named refname could be created without\n+ * conflicting with the name of an existing reference. If\n+ * skip is non-NULL, ignore potential conflicts with refs in skip\n+ * (e.g., because they are scheduled for deletion in the same\n+ * operation).\n+ *\n+ * Two reference names conflict if one of them exactly matches the\n+ * leading components of the other; e.g., \"foo/bar\" conflicts with\n+ * both \"foo\" and with \"foo/bar/baz\" but not with \"foo/bar\" or\n+ * \"foo/barbados\".\n+ *\n+ * skip must be sorted.\n+ */\n+int verify_refname_available(const char *newname, struct string_list *extra,\n+\t\t\t struct string_list *skip, struct strbuf *err);\n+\n+/*\n * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.\n * REF_NODEREF: act on the ref directly, instead of dereferencing\n * symbolic references.\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270844","messageId":"1443477738-32023-3-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 02/43] refs: make repack_without_refs and is_branch public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:37Z","receivedAt":"2015-09-28T22:01:37Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs.c | 9 +--------\n refs.h | 13 +++++++++++++\n 2 files changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex ce551e9..2741cc5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2826,14 +2826,7 @@ int pack_refs(unsigned int flags)\n \treturn 0;\n }\n \n-/*\n- * Rewrite the packed-refs file, omitting any refs listed in\n- * 'refnames'. On error, leave packed-refs unchanged, write an error\n- * message to 'err', and return a nonzero value.\n- *\n- * The refs in 'refnames' needn't be sorted. `err` must not be NULL.\n- */\n-static int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n+int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n {\n \tstruct ref_dir *packed;\n \tstruct string_list_item *refname;\ndiff --git a/refs.h b/refs.h\nindex 79ea220..729bc3c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -218,6 +218,19 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st\n int pack_refs(unsigned int flags);\n \n /*\n+ * Rewrite the packed-refs file, omitting any refs listed in\n+ * 'refnames'. On error, packed-refs will be unchanged, the return\n+ * value is nonzero, and a message about the error is written to the\n+ * 'err' strbuf.\n+ *\n+ * The refs in 'refnames' needn't be sorted. `err` must not be NULL.\n+ */\n+extern int repack_without_refs(struct string_list *refnames,\n+\t\t\t struct strbuf *err);\n+\n+extern int is_branch(const char *refname);\n+\n+/*\n * Return true iff a reference named refname could be created without\n * conflicting with the name of an existing reference. If\n * skip is non-NULL, ignore potential conflicts with refs in skip\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270843","messageId":"1443477738-32023-4-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 03/43] refs-be-files.c: rename refs to refs-be-files","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:38Z","receivedAt":"2015-09-28T22:01:38Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nRename refs.c to refs-be-files.c to indicate that this file now\nholds the implementation for the files based refs backend.\nA smaller portion of the code in this file is backend agnostic and will\nbe moved to a a new refs.c file that will hold all the common refs code\nthat is shared across all backends.\n\nA second reason for first moving all the code to the new file and then\nmove the backend agnostic code back to refs.c instead of the other way\naround is because the code that will eventually remain in this new\nrefs-be-files.c file is so entangled that it would then be very\ndifficult to break the split up into small independent patches/chunks.\n\nSigned-off-by: Ronnie Sahlberg \n---\n Makefile | 2 +-\n refs.c => refs-be-files.c | 0\n 2 files changed, 1 insertion(+), 1 deletion(-)\n rename refs.c => refs-be-files.c (100%)\n\ndiff --git a/Makefile b/Makefile\nindex 46d0ca7..19036de 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -766,7 +766,7 @@ LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n LIB_OBJS += reflog-walk.o\n-LIB_OBJS += refs.o\n+LIB_OBJS += refs-be-files.o\n LIB_OBJS += ref-filter.o\n LIB_OBJS += remote.o\n LIB_OBJS += replace_object.o\ndiff --git a/refs.c b/refs-be-files.c\nsimilarity index 100%\nrename from refs.c\nrename to refs-be-files.c\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270884","messageId":"1443477738-32023-5-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 04/43] refs.c: add a new refs.c file to hold all common refs code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:39Z","receivedAt":"2015-09-28T22:01:39Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nCreate a new refs.c file that will be used to hold all the refs\ncode that is backend agnostic and will be shared across all backends.\n\nSigned-off-by: Ronnie Sahlberg \n---\n Makefile | 1 +\n refs.c | 3 +++\n 2 files changed, 4 insertions(+)\n create mode 100644 refs.c\n\ndiff --git a/Makefile b/Makefile\nindex 19036de..43ceab0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -767,6 +767,7 @@ LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs-be-files.o\n+LIB_OBJS += refs.o\n LIB_OBJS += ref-filter.o\n LIB_OBJS += remote.o\n LIB_OBJS += replace_object.o\ndiff --git a/refs.c b/refs.c\nnew file mode 100644\nindex 0000000..77492ff\n--- /dev/null\n+++ b/refs.c\n@@ -0,0 +1,3 @@\n+/*\n+ * Common refs code for all backends.\n+ */\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270885","messageId":"1443477738-32023-6-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 05/43] refs.c: move update_ref to refs.c","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:40Z","receivedAt":"2015-09-28T22:01:40Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nMove update_ref() to the refs.c file since this function does not\ncontain any backend specific code. Move the ref classifier functions\nas well, since update_ref depends on them.\n\nBased on Ronnie Sahlberg's patch\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 117 +-------------------------------------------------------\n refs.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+), 116 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 2741cc5..2e8079e 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2687,8 +2687,6 @@ struct pack_refs_cb_data {\n \tstruct ref_to_prune *ref_to_prune;\n };\n \n-static int is_per_worktree_ref(const char *refname);\n-\n /*\n * An each_ref_entry_fn that is run over loose references only. If\n * the loose reference can be packed, add an entry in the packed ref\n@@ -2703,7 +2701,7 @@ static int pack_if_possible_fn(struct ref_entry *entry, void *cb_data)\n \tint is_tag_ref = starts_with(entry->name, \"refs/tags/\");\n \n \t/* Do not pack per-worktree refs: */\n-\tif (is_per_worktree_ref(entry->name))\n+\tif (ref_type(entry->name) == REF_TYPE_PER_WORKTREE)\n \t\treturn 0;\n \n \t/* ALWAYS pack tags */\n@@ -2891,77 +2889,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n \treturn 0;\n }\n \n-static int is_per_worktree_ref(const char *refname)\n-{\n-\treturn !strcmp(refname, \"HEAD\") ||\n-\t\tstarts_with(refname, \"refs/bisect/\");\n-}\n-\n-static int is_pseudoref_syntax(const char *refname)\n-{\n-\tconst char *c;\n-\n-\tfor (c = refname; *c; c++) {\n-\t\tif (!isupper(*c) && *c != '-' && *c != '_')\n-\t\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n-enum ref_type ref_type(const char *refname)\n-{\n-\tif (is_per_worktree_ref(refname))\n-\t\treturn REF_TYPE_PER_WORKTREE;\n-\tif (is_pseudoref_syntax(refname))\n-\t\treturn REF_TYPE_PSEUDOREF;\n- return REF_TYPE_NORMAL;\n-}\n-\n-static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,\n-\t\t\t const unsigned char *old_sha1, struct strbuf *err)\n-{\n-\tconst char *filename;\n-\tint fd;\n-\tstatic struct lock_file lock;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tint ret = -1;\n-\n-\tstrbuf_addf(&buf, \"%s\\n\", sha1_to_hex(sha1));\n-\n-\tfilename = git_path(\"%s\", pseudoref);\n-\tfd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);\n-\tif (fd < 0) {\n-\t\tstrbuf_addf(err, \"Could not open '%s' for writing: %s\",\n-\t\t\t filename, strerror(errno));\n-\t\treturn -1;\n-\t}\n-\n-\tif (old_sha1) {\n-\t\tunsigned char actual_old_sha1[20];\n-\n-\t\tif (read_ref(pseudoref, actual_old_sha1))\n-\t\t\tdie(\"could not read ref '%s'\", pseudoref);\n-\t\tif (hashcmp(actual_old_sha1, old_sha1)) {\n-\t\t\tstrbuf_addf(err, \"Unexpected sha1 when writing %s\", pseudoref);\n-\t\t\trollback_lock_file(&lock);\n-\t\t\tgoto done;\n-\t\t}\n-\t}\n-\n-\tif (write_in_full(fd, buf.buf, buf.len) != buf.len) {\n-\t\tstrbuf_addf(err, \"Could not write to '%s'\", filename);\n-\t\trollback_lock_file(&lock);\n-\t\tgoto done;\n-\t}\n-\n-\tcommit_lock_file(&lock);\n-\tret = 0;\n-done:\n-\tstrbuf_release(&buf);\n-\treturn ret;\n-}\n-\n static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)\n {\n \tstatic struct lock_file lock;\n@@ -4110,48 +4037,6 @@ int ref_transaction_verify(struct ref_transaction *transaction,\n \t\t\t\t flags, NULL, err);\n }\n \n-int update_ref(const char *msg, const char *refname,\n-\t const unsigned char *new_sha1, const unsigned char *old_sha1,\n-\t unsigned int flags, enum action_on_err onerr)\n-{\n-\tstruct ref_transaction *t = NULL;\n-\tstruct strbuf err = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tif (ref_type(refname) == REF_TYPE_PSEUDOREF) {\n-\t\tret = write_pseudoref(refname, new_sha1, old_sha1, &err);\n-\t} else {\n-\t\tt = ref_transaction_begin(&err);\n-\t\tif (!t ||\n-\t\t ref_transaction_update(t, refname, new_sha1, old_sha1,\n-\t\t\t\t\t flags, msg, &err) ||\n-\t\t ref_transaction_commit(t, &err)) {\n-\t\t\tret = 1;\n-\t\t\tref_transaction_free(t);\n-\t\t}\n-\t}\n-\tif (ret) {\n-\t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n-\n-\t\tswitch (onerr) {\n-\t\tcase UPDATE_REFS_MSG_ON_ERR:\n-\t\t\terror(str, refname, err.buf);\n-\t\t\tbreak;\n-\t\tcase UPDATE_REFS_DIE_ON_ERR:\n-\t\t\tdie(str, refname, err.buf);\n-\t\t\tbreak;\n-\t\tcase UPDATE_REFS_QUIET_ON_ERR:\n-\t\t\tbreak;\n-\t\t}\n-\t\tstrbuf_release(&err);\n-\t\treturn 1;\n-\t}\n-\tstrbuf_release(&err);\n-\tif (t)\n-\t\tref_transaction_free(t);\n-\treturn 0;\n-}\n-\n static int ref_update_reject_duplicates(struct string_list *refnames,\n \t\t\t\t\tstruct strbuf *err)\n {\ndiff --git a/refs.c b/refs.c\nindex 77492ff..2d10708 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1,3 +1,118 @@\n /*\n * Common refs code for all backends.\n */\n+#include \"cache.h\"\n+#include \"refs.h\"\n+#include \"lockfile.h\"\n+\n+static int is_per_worktree_ref(const char *refname)\n+{\n+\treturn !strcmp(refname, \"HEAD\") ||\n+\t\tstarts_with(refname, \"refs/bisect/\");\n+}\n+\n+static int is_pseudoref_syntax(const char *refname)\n+{\n+\tconst char *c;\n+\n+\tfor (c = refname; *c; c++) {\n+\t\tif (!isupper(*c) && *c != '-' && *c != '_')\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+enum ref_type ref_type(const char *refname)\n+{\n+\tif (is_per_worktree_ref(refname))\n+\t\treturn REF_TYPE_PER_WORKTREE;\n+\tif (is_pseudoref_syntax(refname))\n+\t\treturn REF_TYPE_PSEUDOREF;\n+ return REF_TYPE_NORMAL;\n+}\n+\n+static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,\n+\t\t\t const unsigned char *old_sha1, struct strbuf *err)\n+{\n+\tconst char *filename;\n+\tint fd;\n+\tstatic struct lock_file lock;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret = -1;\n+\n+\tstrbuf_addf(&buf, \"%s\\n\", sha1_to_hex(sha1));\n+\n+\tfilename = git_path(\"%s\", pseudoref);\n+\tfd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);\n+\tif (fd < 0) {\n+\t\tstrbuf_addf(err, \"Could not open '%s' for writing: %s\",\n+\t\t\t filename, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\tif (old_sha1) {\n+\t\tunsigned char actual_old_sha1[20];\n+\t\tread_ref(pseudoref, actual_old_sha1);\n+\t\tif (hashcmp(actual_old_sha1, old_sha1)) {\n+\t\t\tstrbuf_addf(err, \"Unexpected sha1 when writing %s\", pseudoref);\n+\t\t\trollback_lock_file(&lock);\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (write_in_full(fd, buf.buf, buf.len) != buf.len) {\n+\t\tstrbuf_addf(err, \"Could not write to '%s'\", filename);\n+\t\trollback_lock_file(&lock);\n+\t\tgoto done;\n+\t}\n+\n+\tcommit_lock_file(&lock);\n+\tret = 0;\n+done:\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n+int update_ref(const char *msg, const char *refname,\n+\t const unsigned char *new_sha1, const unsigned char *old_sha1,\n+\t unsigned int flags, enum action_on_err onerr)\n+{\n+\tstruct ref_transaction *t = NULL;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (ref_type(refname) == REF_TYPE_PSEUDOREF) {\n+\t\tret = write_pseudoref(refname, new_sha1, old_sha1, &err);\n+\t} else {\n+\t\tt = ref_transaction_begin(&err);\n+\t\tif (!t ||\n+\t\t ref_transaction_update(t, refname, new_sha1, old_sha1,\n+\t\t\t\t\t flags, msg, &err) ||\n+\t\t ref_transaction_commit(t, &err)) {\n+\t\t\tret = 1;\n+\t\t\tref_transaction_free(t);\n+\t\t}\n+\t}\n+\n+\tif (ret) {\n+\t\tconst char *str = \"update_ref failed for ref '%s': %s\";\n+\n+\t\tswitch (onerr) {\n+\t\tcase UPDATE_REFS_MSG_ON_ERR:\n+\t\t\terror(str, refname, err.buf);\n+\t\t\tbreak;\n+\t\tcase UPDATE_REFS_DIE_ON_ERR:\n+\t\t\tdie(str, refname, err.buf);\n+\t\t\tbreak;\n+\t\tcase UPDATE_REFS_QUIET_ON_ERR:\n+\t\t\tbreak;\n+\t\t}\n+\t\tstrbuf_release(&err);\n+\t\treturn 1;\n+\t}\n+\tstrbuf_release(&err);\n+\tif (t)\n+\t\tref_transaction_free(t);\n+\treturn 0;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270881","messageId":"1443477738-32023-7-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 06/43] refs.c: move delete_ref and delete_refs to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:41Z","receivedAt":"2015-09-28T22:01:41Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nMove delete_pseudoref, delete_ref() and delete_refs() to the refs.c\nfile since these functions do not contain any backend specific code.\n\nBased on a patch by Ronnie Sahlberg.\n\nSigned-off-by: David Turner \nSigned-off-by: Ronnie Sahlberg \n---\n refs-be-files.c | 94 ---------------------------------------------------------\n refs.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 94 insertions(+), 94 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 2e8079e..666ba7f 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2889,100 +2889,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)\n \treturn 0;\n }\n \n-static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)\n-{\n-\tstatic struct lock_file lock;\n-\tconst char *filename;\n-\n-\tfilename = git_path(\"%s\", pseudoref);\n-\n-\tif (old_sha1 && !is_null_sha1(old_sha1)) {\n-\t\tint fd;\n-\t\tunsigned char actual_old_sha1[20];\n-\n-\t\tfd = hold_lock_file_for_update(&lock, filename,\n-\t\t\t\t\t LOCK_DIE_ON_ERROR);\n-\t\tif (fd < 0)\n-\t\t\tdie_errno(_(\"Could not open '%s' for writing\"), filename);\n-\t\tif (read_ref(pseudoref, actual_old_sha1))\n-\t\t\tdie(\"could not read ref '%s'\", pseudoref);\n-\t\tif (hashcmp(actual_old_sha1, old_sha1)) {\n-\t\t\twarning(\"Unexpected sha1 when deleting %s\", pseudoref);\n-\t\t\trollback_lock_file(&lock);\n-\t\t\treturn -1;\n-\t\t}\n-\n-\t\tunlink(filename);\n-\t\trollback_lock_file(&lock);\n-\t} else {\n-\t\tunlink(filename);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-int delete_ref(const char *refname, const unsigned char *old_sha1,\n-\t unsigned int flags)\n-{\n-\tstruct ref_transaction *transaction;\n-\tstruct strbuf err = STRBUF_INIT;\n-\n-\tif (ref_type(refname) == REF_TYPE_PSEUDOREF)\n-\t\treturn delete_pseudoref(refname, old_sha1);\n-\n-\ttransaction = ref_transaction_begin(&err);\n-\tif (!transaction ||\n-\t ref_transaction_delete(transaction, refname, old_sha1,\n-\t\t\t\t flags, NULL, &err) ||\n-\t ref_transaction_commit(transaction, &err)) {\n-\t\terror(\"%s\", err.buf);\n-\t\tref_transaction_free(transaction);\n-\t\tstrbuf_release(&err);\n-\t\treturn 1;\n-\t}\n-\tref_transaction_free(transaction);\n-\tstrbuf_release(&err);\n-\treturn 0;\n-}\n-\n-int delete_refs(struct string_list *refnames)\n-{\n-\tstruct strbuf err = STRBUF_INIT;\n-\tint i, result = 0;\n-\n-\tif (!refnames->nr)\n-\t\treturn 0;\n-\n-\tresult = repack_without_refs(refnames, &err);\n-\tif (result) {\n-\t\t/*\n-\t\t * If we failed to rewrite the packed-refs file, then\n-\t\t * it is unsafe to try to remove loose refs, because\n-\t\t * doing so might expose an obsolete packed value for\n-\t\t * a reference that might even point at an object that\n-\t\t * has been garbage collected.\n-\t\t */\n-\t\tif (refnames->nr == 1)\n-\t\t\terror(_(\"could not delete reference %s: %s\"),\n-\t\t\t refnames->items[0].string, err.buf);\n-\t\telse\n-\t\t\terror(_(\"could not delete references: %s\"), err.buf);\n-\n-\t\tgoto out;\n-\t}\n-\n-\tfor (i = 0; i < refnames->nr; i++) {\n-\t\tconst char *refname = refnames->items[i].string;\n-\n-\t\tif (delete_ref(refname, NULL, 0))\n-\t\t\tresult |= error(_(\"could not remove reference %s\"), refname);\n-\t}\n-\n-out:\n-\tstrbuf_release(&err);\n-\treturn result;\n-}\n-\n /*\n * People using contrib's git-new-workdir have .git/logs/refs ->\n * /some/other/path/.git/logs/refs, and that may live on another device.\ndiff --git a/refs.c b/refs.c\nindex 2d10708..205a899 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -116,3 +116,97 @@ int update_ref(const char *msg, const char *refname,\n \t\tref_transaction_free(t);\n \treturn 0;\n }\n+\n+\n+static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)\n+{\n+\tstatic struct lock_file lock;\n+\tconst char *filename;\n+\n+\tfilename = git_path(\"%s\", pseudoref);\n+\n+\tif (old_sha1 && !is_null_sha1(old_sha1)) {\n+\t\tint fd;\n+\t\tunsigned char actual_old_sha1[20];\n+\n+\t\tfd = hold_lock_file_for_update(&lock, filename,\n+\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(_(\"Could not open '%s' for writing\"), filename);\n+\t\tread_ref(pseudoref, actual_old_sha1);\n+\t\tif (hashcmp(actual_old_sha1, old_sha1)) {\n+\t\t\twarning(\"Unexpected sha1 when deleting %s\", pseudoref);\n+\t\t\trollback_lock_file(&lock);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tunlink(filename);\n+\t\trollback_lock_file(&lock);\n+\t} else {\n+\t\tunlink(filename);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int delete_ref(const char *refname, const unsigned char *old_sha1,\n+\t unsigned int flags)\n+{\n+\tstruct ref_transaction *transaction;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tif (ref_type(refname) == REF_TYPE_PSEUDOREF)\n+\t\treturn delete_pseudoref(refname, old_sha1);\n+\n+\ttransaction = ref_transaction_begin(&err);\n+\tif (!transaction ||\n+\t ref_transaction_delete(transaction, refname, old_sha1,\n+\t\t\t\t flags, NULL, &err) ||\n+\t ref_transaction_commit(transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tref_transaction_free(transaction);\n+\t\tstrbuf_release(&err);\n+\t\treturn 1;\n+\t}\n+\tref_transaction_free(transaction);\n+\tstrbuf_release(&err);\n+\treturn 0;\n+}\n+\n+int delete_refs(struct string_list *refnames)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint i, result = 0;\n+\n+\tif (!refnames->nr)\n+\t\treturn 0;\n+\n+\tresult = repack_without_refs(refnames, &err);\n+\tif (result) {\n+\t\t/*\n+\t\t * If we failed to rewrite the packed-refs file, then\n+\t\t * it is unsafe to try to remove loose refs, because\n+\t\t * doing so might expose an obsolete packed value for\n+\t\t * a reference that might even point at an object that\n+\t\t * has been garbage collected.\n+\t\t */\n+\t\tif (refnames->nr == 1)\n+\t\t\terror(_(\"could not delete reference %s: %s\"),\n+\t\t\t refnames->items[0].string, err.buf);\n+\t\telse\n+\t\t\terror(_(\"could not delete references: %s\"), err.buf);\n+\n+\t\tgoto out;\n+\t}\n+\n+\tfor (i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\n+\t\tif (delete_ref(refname, NULL, 0))\n+\t\t\tresult |= error(_(\"could not remove reference %s\"), refname);\n+\t}\n+\n+out:\n+\tstrbuf_release(&err);\n+\treturn result;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270846","messageId":"1443477738-32023-8-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 07/43] refs.c: move read_ref_at to the common refs file","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:42Z","receivedAt":"2015-09-28T22:01:42Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nMove read_ref_at() to the refs.c file since this function does not\ncontain any backend specific code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 118 --------------------------------------------------------\n refs.c | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 118 insertions(+), 118 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 666ba7f..f6cdea9 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3436,124 +3436,6 @@ int create_symref(const char *ref_target, const char *refs_heads_master,\n \treturn 0;\n }\n \n-struct read_ref_at_cb {\n-\tconst char *refname;\n-\tunsigned long at_time;\n-\tint cnt;\n-\tint reccnt;\n-\tunsigned char *sha1;\n-\tint found_it;\n-\n-\tunsigned char osha1[20];\n-\tunsigned char nsha1[20];\n-\tint tz;\n-\tunsigned long date;\n-\tchar **msg;\n-\tunsigned long *cutoff_time;\n-\tint *cutoff_tz;\n-\tint *cutoff_cnt;\n-};\n-\n-static int read_ref_at_ent(unsigned char *osha1, unsigned char *nsha1,\n-\t\tconst char *email, unsigned long timestamp, int tz,\n-\t\tconst char *message, void *cb_data)\n-{\n-\tstruct read_ref_at_cb *cb = cb_data;\n-\n-\tcb->reccnt++;\n-\tcb->tz = tz;\n-\tcb->date = timestamp;\n-\n-\tif (timestamp <= cb->at_time || cb->cnt == 0) {\n-\t\tif (cb->msg)\n-\t\t\t*cb->msg = xstrdup(message);\n-\t\tif (cb->cutoff_time)\n-\t\t\t*cb->cutoff_time = timestamp;\n-\t\tif (cb->cutoff_tz)\n-\t\t\t*cb->cutoff_tz = tz;\n-\t\tif (cb->cutoff_cnt)\n-\t\t\t*cb->cutoff_cnt = cb->reccnt - 1;\n-\t\t/*\n-\t\t * we have not yet updated cb->[n|o]sha1 so they still\n-\t\t * hold the values for the previous record.\n-\t\t */\n-\t\tif (!is_null_sha1(cb->osha1)) {\n-\t\t\thashcpy(cb->sha1, nsha1);\n-\t\t\tif (hashcmp(cb->osha1, nsha1))\n-\t\t\t\twarning(\"Log for ref %s has gap after %s.\",\n-\t\t\t\t\tcb->refname, show_date(cb->date, cb->tz, DATE_MODE(RFC2822)));\n-\t\t}\n-\t\telse if (cb->date == cb->at_time)\n-\t\t\thashcpy(cb->sha1, nsha1);\n-\t\telse if (hashcmp(nsha1, cb->sha1))\n-\t\t\twarning(\"Log for ref %s unexpectedly ended on %s.\",\n-\t\t\t\tcb->refname, show_date(cb->date, cb->tz,\n-\t\t\t\t\t\t DATE_MODE(RFC2822)));\n-\t\thashcpy(cb->osha1, osha1);\n-\t\thashcpy(cb->nsha1, nsha1);\n-\t\tcb->found_it = 1;\n-\t\treturn 1;\n-\t}\n-\thashcpy(cb->osha1, osha1);\n-\thashcpy(cb->nsha1, nsha1);\n-\tif (cb->cnt > 0)\n-\t\tcb->cnt--;\n-\treturn 0;\n-}\n-\n-static int read_ref_at_ent_oldest(unsigned char *osha1, unsigned char *nsha1,\n-\t\t\t\t const char *email, unsigned long timestamp,\n-\t\t\t\t int tz, const char *message, void *cb_data)\n-{\n-\tstruct read_ref_at_cb *cb = cb_data;\n-\n-\tif (cb->msg)\n-\t\t*cb->msg = xstrdup(message);\n-\tif (cb->cutoff_time)\n-\t\t*cb->cutoff_time = timestamp;\n-\tif (cb->cutoff_tz)\n-\t\t*cb->cutoff_tz = tz;\n-\tif (cb->cutoff_cnt)\n-\t\t*cb->cutoff_cnt = cb->reccnt;\n-\thashcpy(cb->sha1, osha1);\n-\tif (is_null_sha1(cb->sha1))\n-\t\thashcpy(cb->sha1, nsha1);\n-\t/* We just want the first entry */\n-\treturn 1;\n-}\n-\n-int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time, int cnt,\n-\t\tunsigned char *sha1, char **msg,\n-\t\tunsigned long *cutoff_time, int *cutoff_tz, int *cutoff_cnt)\n-{\n-\tstruct read_ref_at_cb cb;\n-\n-\tmemset(&cb, 0, sizeof(cb));\n-\tcb.refname = refname;\n-\tcb.at_time = at_time;\n-\tcb.cnt = cnt;\n-\tcb.msg = msg;\n-\tcb.cutoff_time = cutoff_time;\n-\tcb.cutoff_tz = cutoff_tz;\n-\tcb.cutoff_cnt = cutoff_cnt;\n-\tcb.sha1 = sha1;\n-\n-\tfor_each_reflog_ent_reverse(refname, read_ref_at_ent, &cb);\n-\n-\tif (!cb.reccnt) {\n-\t\tif (flags & GET_SHA1_QUIETLY)\n-\t\t\texit(128);\n-\t\telse\n-\t\t\tdie(\"Log for %s is empty.\", refname);\n-\t}\n-\tif (cb.found_it)\n-\t\treturn 0;\n-\n-\tfor_each_reflog_ent(refname, read_ref_at_ent_oldest, &cb);\n-\n-\treturn 1;\n-}\n-\n int reflog_exists(const char *refname)\n {\n \tstruct stat st;\ndiff --git a/refs.c b/refs.c\nindex 205a899..5a8ef0c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -210,3 +210,121 @@ out:\n \tstrbuf_release(&err);\n \treturn result;\n }\n+\n+struct read_ref_at_cb {\n+\tconst char *refname;\n+\tunsigned long at_time;\n+\tint cnt;\n+\tint reccnt;\n+\tunsigned char *sha1;\n+\tint found_it;\n+\n+\tunsigned char osha1[20];\n+\tunsigned char nsha1[20];\n+\tint tz;\n+\tunsigned long date;\n+\tchar **msg;\n+\tunsigned long *cutoff_time;\n+\tint *cutoff_tz;\n+\tint *cutoff_cnt;\n+};\n+\n+static int read_ref_at_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\tconst char *email, unsigned long timestamp, int tz,\n+\t\tconst char *message, void *cb_data)\n+{\n+\tstruct read_ref_at_cb *cb = cb_data;\n+\n+\tcb->reccnt++;\n+\tcb->tz = tz;\n+\tcb->date = timestamp;\n+\n+\tif (timestamp <= cb->at_time || cb->cnt == 0) {\n+\t\tif (cb->msg)\n+\t\t\t*cb->msg = xstrdup(message);\n+\t\tif (cb->cutoff_time)\n+\t\t\t*cb->cutoff_time = timestamp;\n+\t\tif (cb->cutoff_tz)\n+\t\t\t*cb->cutoff_tz = tz;\n+\t\tif (cb->cutoff_cnt)\n+\t\t\t*cb->cutoff_cnt = cb->reccnt - 1;\n+\t\t/*\n+\t\t * we have not yet updated cb->[n|o]sha1 so they still\n+\t\t * hold the values for the previous record.\n+\t\t */\n+\t\tif (!is_null_sha1(cb->osha1)) {\n+\t\t\thashcpy(cb->sha1, nsha1);\n+\t\t\tif (hashcmp(cb->osha1, nsha1))\n+\t\t\t\twarning(\"Log for ref %s has gap after %s.\",\n+\t\t\t\t\tcb->refname, show_date(cb->date, cb->tz, DATE_MODE(RFC2822)));\n+\t\t}\n+\t\telse if (cb->date == cb->at_time)\n+\t\t\thashcpy(cb->sha1, nsha1);\n+\t\telse if (hashcmp(nsha1, cb->sha1))\n+\t\t\twarning(\"Log for ref %s unexpectedly ended on %s.\",\n+\t\t\t\tcb->refname, show_date(cb->date, cb->tz,\n+\t\t\t\t\t\t DATE_MODE(RFC2822)));\n+\t\thashcpy(cb->osha1, osha1);\n+\t\thashcpy(cb->nsha1, nsha1);\n+\t\tcb->found_it = 1;\n+\t\treturn 1;\n+\t}\n+\thashcpy(cb->osha1, osha1);\n+\thashcpy(cb->nsha1, nsha1);\n+\tif (cb->cnt > 0)\n+\t\tcb->cnt--;\n+\treturn 0;\n+}\n+\n+static int read_ref_at_ent_oldest(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t\t const char *email, unsigned long timestamp,\n+\t\t\t\t int tz, const char *message, void *cb_data)\n+{\n+\tstruct read_ref_at_cb *cb = cb_data;\n+\n+\tif (cb->msg)\n+\t\t*cb->msg = xstrdup(message);\n+\tif (cb->cutoff_time)\n+\t\t*cb->cutoff_time = timestamp;\n+\tif (cb->cutoff_tz)\n+\t\t*cb->cutoff_tz = tz;\n+\tif (cb->cutoff_cnt)\n+\t\t*cb->cutoff_cnt = cb->reccnt;\n+\thashcpy(cb->sha1, osha1);\n+\tif (is_null_sha1(cb->sha1))\n+\t\thashcpy(cb->sha1, nsha1);\n+\t/* We just want the first entry */\n+\treturn 1;\n+}\n+\n+int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time, int cnt,\n+\t\tunsigned char *sha1, char **msg,\n+\t\tunsigned long *cutoff_time, int *cutoff_tz, int *cutoff_cnt)\n+{\n+\tstruct read_ref_at_cb cb;\n+\n+\tmemset(&cb, 0, sizeof(cb));\n+\tcb.refname = refname;\n+\tcb.at_time = at_time;\n+\tcb.cnt = cnt;\n+\tcb.msg = msg;\n+\tcb.cutoff_time = cutoff_time;\n+\tcb.cutoff_tz = cutoff_tz;\n+\tcb.cutoff_cnt = cutoff_cnt;\n+\tcb.sha1 = sha1;\n+\n+\tfor_each_reflog_ent_reverse(refname, read_ref_at_ent, &cb);\n+\n+\tif (!cb.reccnt) {\n+\t\tif (flags & GET_SHA1_QUIETLY)\n+\t\t\texit(128);\n+\t\telse\n+\t\t\tdie(\"Log for %s is empty.\", refname);\n+\t}\n+\tif (cb.found_it)\n+\t\treturn 0;\n+\n+\tfor_each_reflog_ent(refname, read_ref_at_ent_oldest, &cb);\n+\n+\treturn 1;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270882","messageId":"1443477738-32023-9-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 08/43] refs.c: move the hidden refs functions to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:43Z","receivedAt":"2015-09-28T22:01:43Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nMove the hidden refs functions to the refs.c file since these\nfunctions do not contain any backend specific code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 51 ---------------------------------------------------\n refs.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 51 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex f6cdea9..3609d02 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -4181,57 +4181,6 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \treturn xstrdup(refname);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n-{\n-\tif (!strcmp(\"transfer.hiderefs\", var) ||\n-\t /* NEEDSWORK: use parse_config_key() once both are merged */\n-\t (starts_with(var, section) && var[strlen(section)] == '.' &&\n-\t !strcmp(var + strlen(section), \".hiderefs\"))) {\n-\t\tchar *ref;\n-\t\tint len;\n-\n-\t\tif (!value)\n-\t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n-\t\tlen = strlen(ref);\n-\t\twhile (len && ref[len - 1] == '/')\n-\t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\thide_refs = xcalloc(1, sizeof(*hide_refs));\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n-\t\tstring_list_append(hide_refs, ref);\n-\t}\n-\treturn 0;\n-}\n-\n-int ref_is_hidden(const char *refname)\n-{\n-\tint i;\n-\n-\tif (!hide_refs)\n-\t\treturn 0;\n-\tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n-\t\tint neg = 0;\n-\t\tint len;\n-\n-\t\tif (*match == '!') {\n-\t\t\tneg = 1;\n-\t\t\tmatch++;\n-\t\t}\n-\n-\t\tif (!starts_with(refname, match))\n-\t\t\tcontinue;\n-\t\tlen = strlen(match);\n-\t\tif (!refname[len] || refname[len] == '/')\n-\t\t\treturn !neg;\n-\t}\n-\treturn 0;\n-}\n-\n struct expire_reflog_cb {\n \tunsigned int flags;\n \treflog_expiry_should_prune_fn *should_prune_fn;\ndiff --git a/refs.c b/refs.c\nindex 5a8ef0c..6b2fc39 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -328,3 +328,53 @@ int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time,\n \n \treturn 1;\n }\n+\n+static struct string_list *hide_refs;\n+\n+int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+{\n+\tif (!strcmp(\"transfer.hiderefs\", var) ||\n+\t /* NEEDSWORK: use parse_config_key() once both are merged */\n+\t (starts_with(var, section) && var[strlen(section)] == '.' &&\n+\t !strcmp(var + strlen(section), \".hiderefs\"))) {\n+\t\tchar *ref;\n+\t\tint len;\n+\n+\t\tif (!value)\n+\t\t\treturn config_error_nonbool(var);\n+\t\tref = xstrdup(value);\n+\t\tlen = strlen(ref);\n+\t\twhile (len && ref[len - 1] == '/')\n+\t\t\tref[--len] = '\\0';\n+\t\tif (!hide_refs) {\n+\t\t\thide_refs = xcalloc(1, sizeof(*hide_refs));\n+\t\t\thide_refs->strdup_strings = 1;\n+\t\t}\n+\t\tstring_list_append(hide_refs, ref);\n+\t}\n+\treturn 0;\n+}\n+\n+int ref_is_hidden(const char *refname)\n+{\n+\tint i;\n+\n+\tif (!hide_refs)\n+\t\treturn 0;\n+\tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n+\t\tconst char *match = hide_refs->items[i].string;\n+\t\tint neg = 0;\n+\t\tint len;\n+\t\tif (*match == '!') {\n+\t\t\tneg = 1;\n+\t\t\tmatch++;\n+\t\t}\n+\n+\t\tif (!starts_with(refname, match))\n+\t\t\tcontinue;\n+\t\tlen = strlen(match);\n+\t\tif (!refname[len] || refname[len] == '/')\n+\t\t\treturn !neg;\n+\t}\n+\treturn 0;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270847","messageId":"1443477738-32023-10-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 09/43] refs.c: move dwim and friend functions to the common refs code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:44Z","receivedAt":"2015-09-28T22:01:44Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nThese functions do not contain any backend specific code so we move\nthem to the common code and share across all backends.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 203 --------------------------------------------------------\n refs.c | 203 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 203 insertions(+), 203 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 3609d02..eb660bb 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2259,30 +2259,6 @@ const char *prettify_refname(const char *name)\n \t\t0);\n }\n \n-static const char *ref_rev_parse_rules[] = {\n-\t\"%.*s\",\n-\t\"refs/%.*s\",\n-\t\"refs/tags/%.*s\",\n-\t\"refs/heads/%.*s\",\n-\t\"refs/remotes/%.*s\",\n-\t\"refs/remotes/%.*s/HEAD\",\n-\tNULL\n-};\n-\n-int refname_match(const char *abbrev_name, const char *full_name)\n-{\n-\tconst char **p;\n-\tconst int abbrev_name_len = strlen(abbrev_name);\n-\n-\tfor (p = ref_rev_parse_rules; *p; p++) {\n-\t\tif (!strcmp(full_name, mkpath(*p, abbrev_name_len, abbrev_name))) {\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\n-\treturn 0;\n-}\n-\n static void unlock_ref(struct ref_lock *lock)\n {\n \t/* Do not free lock->lk -- atexit() still looks at them */\n@@ -2335,92 +2311,6 @@ static int remove_empty_directories(struct strbuf *path)\n }\n \n /*\n- * *string and *len will only be substituted, and *string returned (for\n- * later free()ing) if the string passed in is a magic short-hand form\n- * to name a branch.\n- */\n-static char *substitute_branch_name(const char **string, int *len)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tint ret = interpret_branch_name(*string, *len, &buf);\n-\n-\tif (ret == *len) {\n-\t\tsize_t size;\n-\t\t*string = strbuf_detach(&buf, &size);\n-\t\t*len = size;\n-\t\treturn (char *)*string;\n-\t}\n-\n-\treturn NULL;\n-}\n-\n-int dwim_ref(const char *str, int len, unsigned char *sha1, char **ref)\n-{\n-\tchar *last_branch = substitute_branch_name(&str, &len);\n-\tconst char **p, *r;\n-\tint refs_found = 0;\n-\n-\t*ref = NULL;\n-\tfor (p = ref_rev_parse_rules; *p; p++) {\n-\t\tchar fullref[PATH_MAX];\n-\t\tunsigned char sha1_from_ref[20];\n-\t\tunsigned char *this_result;\n-\t\tint flag;\n-\n-\t\tthis_result = refs_found ? sha1_from_ref : sha1;\n-\t\tmksnpath(fullref, sizeof(fullref), *p, len, str);\n-\t\tr = resolve_ref_unsafe(fullref, RESOLVE_REF_READING,\n-\t\t\t\t this_result, &flag);\n-\t\tif (r) {\n-\t\t\tif (!refs_found++)\n-\t\t\t\t*ref = xstrdup(r);\n-\t\t\tif (!warn_ambiguous_refs)\n-\t\t\t\tbreak;\n-\t\t} else if ((flag & REF_ISSYMREF) && strcmp(fullref, \"HEAD\")) {\n-\t\t\twarning(\"ignoring dangling symref %s.\", fullref);\n-\t\t} else if ((flag & REF_ISBROKEN) && strchr(fullref, '/')) {\n-\t\t\twarning(\"ignoring broken ref %s.\", fullref);\n-\t\t}\n-\t}\n-\tfree(last_branch);\n-\treturn refs_found;\n-}\n-\n-int dwim_log(const char *str, int len, unsigned char *sha1, char **log)\n-{\n-\tchar *last_branch = substitute_branch_name(&str, &len);\n-\tconst char **p;\n-\tint logs_found = 0;\n-\n-\t*log = NULL;\n-\tfor (p = ref_rev_parse_rules; *p; p++) {\n-\t\tunsigned char hash[20];\n-\t\tchar path[PATH_MAX];\n-\t\tconst char *ref, *it;\n-\n-\t\tmksnpath(path, sizeof(path), *p, len, str);\n-\t\tref = resolve_ref_unsafe(path, RESOLVE_REF_READING,\n-\t\t\t\t\t hash, NULL);\n-\t\tif (!ref)\n-\t\t\tcontinue;\n-\t\tif (reflog_exists(path))\n-\t\t\tit = path;\n-\t\telse if (strcmp(ref, path) && reflog_exists(ref))\n-\t\t\tit = ref;\n-\t\telse\n-\t\t\tcontinue;\n-\t\tif (!logs_found++) {\n-\t\t\t*log = xstrdup(it);\n-\t\t\thashcpy(sha1, hash);\n-\t\t}\n-\t\tif (!warn_ambiguous_refs)\n-\t\t\tbreak;\n-\t}\n-\tfree(last_branch);\n-\treturn logs_found;\n-}\n-\n-/*\n * Locks a ref returning the lock on success and NULL on failure.\n * On failure errno is set to something meaningful.\n */\n@@ -4088,99 +3978,6 @@ cleanup:\n \treturn ret;\n }\n \n-char *shorten_unambiguous_ref(const char *refname, int strict)\n-{\n-\tint i;\n-\tstatic char **scanf_fmts;\n-\tstatic int nr_rules;\n-\tchar *short_name;\n-\n-\tif (!nr_rules) {\n-\t\t/*\n-\t\t * Pre-generate scanf formats from ref_rev_parse_rules[].\n-\t\t * Generate a format suitable for scanf from a\n-\t\t * ref_rev_parse_rules rule by interpolating \"%s\" at the\n-\t\t * location of the \"%.*s\".\n-\t\t */\n-\t\tsize_t total_len = 0;\n-\t\tsize_t offset = 0;\n-\n-\t\t/* the rule list is NULL terminated, count them first */\n-\t\tfor (nr_rules = 0; ref_rev_parse_rules[nr_rules]; nr_rules++)\n-\t\t\t/* -2 for strlen(\"%.*s\") - strlen(\"%s\"); +1 for NUL */\n-\t\t\ttotal_len += strlen(ref_rev_parse_rules[nr_rules]) - 2 + 1;\n-\n-\t\tscanf_fmts = xmalloc(nr_rules * sizeof(char *) + total_len);\n-\n-\t\toffset = 0;\n-\t\tfor (i = 0; i < nr_rules; i++) {\n-\t\t\tassert(offset < total_len);\n-\t\t\tscanf_fmts[i] = (char *)&scanf_fmts[nr_rules] + offset;\n-\t\t\toffset += snprintf(scanf_fmts[i], total_len - offset,\n-\t\t\t\t\t ref_rev_parse_rules[i], 2, \"%s\") + 1;\n-\t\t}\n-\t}\n-\n-\t/* bail out if there are no rules */\n-\tif (!nr_rules)\n-\t\treturn xstrdup(refname);\n-\n-\t/* buffer for scanf result, at most refname must fit */\n-\tshort_name = xstrdup(refname);\n-\n-\t/* skip first rule, it will always match */\n-\tfor (i = nr_rules - 1; i > 0 ; --i) {\n-\t\tint j;\n-\t\tint rules_to_fail = i;\n-\t\tint short_name_len;\n-\n-\t\tif (1 != sscanf(refname, scanf_fmts[i], short_name))\n-\t\t\tcontinue;\n-\n-\t\tshort_name_len = strlen(short_name);\n-\n-\t\t/*\n-\t\t * in strict mode, all (except the matched one) rules\n-\t\t * must fail to resolve to a valid non-ambiguous ref\n-\t\t */\n-\t\tif (strict)\n-\t\t\trules_to_fail = nr_rules;\n-\n-\t\t/*\n-\t\t * check if the short name resolves to a valid ref,\n-\t\t * but use only rules prior to the matched one\n-\t\t */\n-\t\tfor (j = 0; j < rules_to_fail; j++) {\n-\t\t\tconst char *rule = ref_rev_parse_rules[j];\n-\t\t\tchar refname[PATH_MAX];\n-\n-\t\t\t/* skip matched rule */\n-\t\t\tif (i == j)\n-\t\t\t\tcontinue;\n-\n-\t\t\t/*\n-\t\t\t * the short name is ambiguous, if it resolves\n-\t\t\t * (with this previous rule) to a valid ref\n-\t\t\t * read_ref() returns 0 on success\n-\t\t\t */\n-\t\t\tmksnpath(refname, sizeof(refname),\n-\t\t\t\t rule, short_name_len, short_name);\n-\t\t\tif (ref_exists(refname))\n-\t\t\t\tbreak;\n-\t\t}\n-\n-\t\t/*\n-\t\t * short name is non-ambiguous if all previous rules\n-\t\t * haven't resolved to a valid ref\n-\t\t */\n-\t\tif (j == rules_to_fail)\n-\t\t\treturn short_name;\n-\t}\n-\n-\tfree(short_name);\n-\treturn xstrdup(refname);\n-}\n-\n struct expire_reflog_cb {\n \tunsigned int flags;\n \treflog_expiry_should_prune_fn *should_prune_fn;\ndiff --git a/refs.c b/refs.c\nindex 6b2fc39..310b7f5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -378,3 +378,206 @@ int ref_is_hidden(const char *refname)\n \t}\n \treturn 0;\n }\n+\n+static const char *ref_rev_parse_rules[] = {\n+\t\"%.*s\",\n+\t\"refs/%.*s\",\n+\t\"refs/tags/%.*s\",\n+\t\"refs/heads/%.*s\",\n+\t\"refs/remotes/%.*s\",\n+\t\"refs/remotes/%.*s/HEAD\",\n+\tNULL\n+};\n+\n+int refname_match(const char *abbrev_name, const char *full_name)\n+{\n+\tconst char **p;\n+\tconst int abbrev_name_len = strlen(abbrev_name);\n+\n+\tfor (p = ref_rev_parse_rules; *p; p++) {\n+\t\tif (!strcmp(full_name, mkpath(*p, abbrev_name_len, abbrev_name))) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * *string and *len will only be substituted, and *string returned (for\n+ * later free()ing) if the string passed in is a magic short-hand form\n+ * to name a branch.\n+ */\n+static char *substitute_branch_name(const char **string, int *len)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret = interpret_branch_name(*string, *len, &buf);\n+\n+\tif (ret == *len) {\n+\t\tsize_t size;\n+\t\t*string = strbuf_detach(&buf, &size);\n+\t\t*len = size;\n+\t\treturn (char *)*string;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+int dwim_ref(const char *str, int len, unsigned char *sha1, char **ref)\n+{\n+\tchar *last_branch = substitute_branch_name(&str, &len);\n+\tconst char **p, *r;\n+\tint refs_found = 0;\n+\n+\t*ref = NULL;\n+\tfor (p = ref_rev_parse_rules; *p; p++) {\n+\t\tchar fullref[PATH_MAX];\n+\t\tunsigned char sha1_from_ref[20];\n+\t\tunsigned char *this_result;\n+\t\tint flag;\n+\n+\t\tthis_result = refs_found ? sha1_from_ref : sha1;\n+\t\tmksnpath(fullref, sizeof(fullref), *p, len, str);\n+\t\tr = resolve_ref_unsafe(fullref, RESOLVE_REF_READING,\n+\t\t\t\t this_result, &flag);\n+\t\tif (r) {\n+\t\t\tif (!refs_found++)\n+\t\t\t\t*ref = xstrdup(r);\n+\t\t\tif (!warn_ambiguous_refs)\n+\t\t\t\tbreak;\n+\t\t} else if ((flag & REF_ISSYMREF) && strcmp(fullref, \"HEAD\")) {\n+\t\t\twarning(\"ignoring dangling symref %s.\", fullref);\n+\t\t} else if ((flag & REF_ISBROKEN) && strchr(fullref, '/')) {\n+\t\t\twarning(\"ignoring broken ref %s.\", fullref);\n+\t\t}\n+\t}\n+\tfree(last_branch);\n+\treturn refs_found;\n+}\n+\n+int dwim_log(const char *str, int len, unsigned char *sha1, char **log)\n+{\n+\tchar *last_branch = substitute_branch_name(&str, &len);\n+\tconst char **p;\n+\tint logs_found = 0;\n+\n+\t*log = NULL;\n+\tfor (p = ref_rev_parse_rules; *p; p++) {\n+\t\tunsigned char hash[20];\n+\t\tchar path[PATH_MAX];\n+\t\tconst char *ref, *it;\n+\n+\t\tmksnpath(path, sizeof(path), *p, len, str);\n+\t\tref = resolve_ref_unsafe(path, RESOLVE_REF_READING,\n+\t\t\t\t\t hash, NULL);\n+\t\tif (!ref)\n+\t\t\tcontinue;\n+\t\tif (reflog_exists(path))\n+\t\t\tit = path;\n+\t\telse if (strcmp(ref, path) && reflog_exists(ref))\n+\t\t\tit = ref;\n+\t\telse\n+\t\t\tcontinue;\n+\t\tif (!logs_found++) {\n+\t\t\t*log = xstrdup(it);\n+\t\t\thashcpy(sha1, hash);\n+\t\t}\n+\t\tif (!warn_ambiguous_refs)\n+\t\t\tbreak;\n+\t}\n+\tfree(last_branch);\n+\treturn logs_found;\n+}\n+\n+char *shorten_unambiguous_ref(const char *refname, int strict)\n+{\n+\tint i;\n+\tstatic char **scanf_fmts;\n+\tstatic int nr_rules;\n+\tchar *short_name;\n+\n+\tif (!nr_rules) {\n+\t\t/*\n+\t\t * Pre-generate scanf formats from ref_rev_parse_rules[].\n+\t\t * Generate a format suitable for scanf from a\n+\t\t * ref_rev_parse_rules rule by interpolating \"%s\" at the\n+\t\t * location of the \"%.*s\".\n+\t\t */\n+\t\tsize_t total_len = 0;\n+\t\tsize_t offset = 0;\n+\n+\t\t/* the rule list is NULL terminated, count them first */\n+\t\tfor (nr_rules = 0; ref_rev_parse_rules[nr_rules]; nr_rules++)\n+\t\t\t/* -2 for strlen(\"%.*s\") - strlen(\"%s\"); +1 for NUL */\n+\t\t\ttotal_len += strlen(ref_rev_parse_rules[nr_rules]) - 2 + 1;\n+\n+\t\tscanf_fmts = xmalloc(nr_rules * sizeof(char *) + total_len);\n+\n+\t\toffset = 0;\n+\t\tfor (i = 0; i < nr_rules; i++) {\n+\t\t\tassert(offset < total_len);\n+\t\t\tscanf_fmts[i] = (char *)&scanf_fmts[nr_rules] + offset;\n+\t\t\toffset += snprintf(scanf_fmts[i], total_len - offset,\n+\t\t\t\t\t ref_rev_parse_rules[i], 2, \"%s\") + 1;\n+\t\t}\n+\t}\n+\n+\t/* bail out if there are no rules */\n+\tif (!nr_rules)\n+\t\treturn xstrdup(refname);\n+\n+\t/* buffer for scanf result, at most refname must fit */\n+\tshort_name = xstrdup(refname);\n+\n+\t/* skip first rule, it will always match */\n+\tfor (i = nr_rules - 1; i > 0 ; --i) {\n+\t\tint j;\n+\t\tint rules_to_fail = i;\n+\t\tint short_name_len;\n+\n+\t\tif (1 != sscanf(refname, scanf_fmts[i], short_name))\n+\t\t\tcontinue;\n+\n+\t\tshort_name_len = strlen(short_name);\n+\n+\t\t/*\n+\t\t * in strict mode, all (except the matched one) rules\n+\t\t * must fail to resolve to a valid non-ambiguous ref\n+\t\t */\n+\t\tif (strict)\n+\t\t\trules_to_fail = nr_rules;\n+\n+\t\t/*\n+\t\t * check if the short name resolves to a valid ref,\n+\t\t * but use only rules prior to the matched one\n+\t\t */\n+\t\tfor (j = 0; j < rules_to_fail; j++) {\n+\t\t\tconst char *rule = ref_rev_parse_rules[j];\n+\t\t\tchar refname[PATH_MAX];\n+\n+\t\t\t/* skip matched rule */\n+\t\t\tif (i == j)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * the short name is ambiguous, if it resolves\n+\t\t\t * (with this previous rule) to a valid ref\n+\t\t\t * read_ref() returns 0 on success\n+\t\t\t */\n+\t\t\tmksnpath(refname, sizeof(refname),\n+\t\t\t\t rule, short_name_len, short_name);\n+\t\t\tif (ref_exists(refname))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\t/*\n+\t\t * short name is non-ambiguous if all previous rules\n+\t\t * haven't resolved to a valid ref\n+\t\t */\n+\t\tif (j == rules_to_fail)\n+\t\t\treturn short_name;\n+\t}\n+\n+\tfree(short_name);\n+\treturn xstrdup(refname);\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270880","messageId":"1443477738-32023-11-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 10/43] refs.c: move warn_if_dangling_symref* to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:45Z","receivedAt":"2015-09-28T22:01:45Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nThese functions do not use any backend specific code so we move\nthem to the common code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 52 ----------------------------------------------------\n refs.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 52 insertions(+), 52 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex eb660bb..2495d42 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1956,58 +1956,6 @@ int peel_ref(const char *refname, unsigned char *sha1)\n \treturn peel_object(base, sha1);\n }\n \n-struct warn_if_dangling_data {\n-\tFILE *fp;\n-\tconst char *refname;\n-\tconst struct string_list *refnames;\n-\tconst char *msg_fmt;\n-};\n-\n-static int warn_if_dangling_symref(const char *refname, const struct object_id *oid,\n-\t\t\t\t int flags, void *cb_data)\n-{\n-\tstruct warn_if_dangling_data *d = cb_data;\n-\tconst char *resolves_to;\n-\tstruct object_id junk;\n-\n-\tif (!(flags & REF_ISSYMREF))\n-\t\treturn 0;\n-\n-\tresolves_to = resolve_ref_unsafe(refname, 0, junk.hash, NULL);\n-\tif (!resolves_to\n-\t || (d->refname\n-\t\t? strcmp(resolves_to, d->refname)\n-\t\t: !string_list_has_string(d->refnames, resolves_to))) {\n-\t\treturn 0;\n-\t}\n-\n-\tfprintf(d->fp, d->msg_fmt, refname);\n-\tfputc('\\n', d->fp);\n-\treturn 0;\n-}\n-\n-void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)\n-{\n-\tstruct warn_if_dangling_data data;\n-\n-\tdata.fp = fp;\n-\tdata.refname = refname;\n-\tdata.refnames = NULL;\n-\tdata.msg_fmt = msg_fmt;\n-\tfor_each_rawref(warn_if_dangling_symref, &data);\n-}\n-\n-void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_list *refnames)\n-{\n-\tstruct warn_if_dangling_data data;\n-\n-\tdata.fp = fp;\n-\tdata.refname = NULL;\n-\tdata.refnames = refnames;\n-\tdata.msg_fmt = msg_fmt;\n-\tfor_each_rawref(warn_if_dangling_symref, &data);\n-}\n-\n /*\n * Call fn for each reference in the specified ref_cache, omitting\n * references not in the containing_dir of base. fn is called for all\ndiff --git a/refs.c b/refs.c\nindex 310b7f5..24d5e28 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -581,3 +581,55 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \tfree(short_name);\n \treturn xstrdup(refname);\n }\n+\n+struct warn_if_dangling_data {\n+\tFILE *fp;\n+\tconst char *refname;\n+\tconst struct string_list *refnames;\n+\tconst char *msg_fmt;\n+};\n+\n+static int warn_if_dangling_symref(const char *refname, const struct object_id *oid,\n+\t\t\t\t int flags, void *cb_data)\n+{\n+\tstruct warn_if_dangling_data *d = cb_data;\n+\tconst char *resolves_to;\n+\tstruct object_id junk;\n+\n+\tif (!(flags & REF_ISSYMREF))\n+\t\treturn 0;\n+\n+\tresolves_to = resolve_ref_unsafe(refname, 0, junk.hash, NULL);\n+\tif (!resolves_to\n+\t || (d->refname\n+\t\t? strcmp(resolves_to, d->refname)\n+\t\t: !string_list_has_string(d->refnames, resolves_to))) {\n+\t\treturn 0;\n+\t}\n+\n+\tfprintf(d->fp, d->msg_fmt, refname);\n+\tfputc('\\n', d->fp);\n+\treturn 0;\n+}\n+\n+void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)\n+{\n+\tstruct warn_if_dangling_data data;\n+\n+\tdata.fp = fp;\n+\tdata.refname = refname;\n+\tdata.refnames = NULL;\n+\tdata.msg_fmt = msg_fmt;\n+\tfor_each_rawref(warn_if_dangling_symref, &data);\n+}\n+\n+void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_list *refnames)\n+{\n+\tstruct warn_if_dangling_data data;\n+\n+\tdata.fp = fp;\n+\tdata.refname = NULL;\n+\tdata.refnames = refnames;\n+\tdata.msg_fmt = msg_fmt;\n+\tfor_each_rawref(warn_if_dangling_symref, &data);\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270878","messageId":"1443477738-32023-12-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 11/43] refs.c: move read_ref, read_ref_full and ref_exists to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:46Z","receivedAt":"2015-09-28T22:01:46Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nThese functions do not depend on the backend implementation so we\nmove them to the common code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 18 ------------------\n refs.c | 18 ++++++++++++++++++\n 2 files changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 2495d42..d46474a 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1803,24 +1803,6 @@ struct ref_filter {\n \tvoid *cb_data;\n };\n \n-int read_ref_full(const char *refname, int resolve_flags, unsigned char *sha1, int *flags)\n-{\n-\tif (resolve_ref_unsafe(refname, resolve_flags, sha1, flags))\n-\t\treturn 0;\n-\treturn -1;\n-}\n-\n-int read_ref(const char *refname, unsigned char *sha1)\n-{\n-\treturn read_ref_full(refname, RESOLVE_REF_READING, sha1, NULL);\n-}\n-\n-int ref_exists(const char *refname)\n-{\n-\tunsigned char sha1[20];\n-\treturn !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);\n-}\n-\n static int filter_refs(const char *refname, const struct object_id *oid,\n \t\t\t int flags, void *data)\n {\ndiff --git a/refs.c b/refs.c\nindex 24d5e28..242f66d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -633,3 +633,21 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li\n \tdata.msg_fmt = msg_fmt;\n \tfor_each_rawref(warn_if_dangling_symref, &data);\n }\n+\n+int read_ref_full(const char *refname, int resolve_flags, unsigned char *sha1, int *flags)\n+{\n+\tif (resolve_ref_unsafe(refname, resolve_flags, sha1, flags))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n+int read_ref(const char *refname, unsigned char *sha1)\n+{\n+\treturn read_ref_full(refname, RESOLVE_REF_READING, sha1, NULL);\n+}\n+\n+int ref_exists(const char *refname)\n+{\n+\tunsigned char sha1[20];\n+\treturn !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270883","messageId":"1443477738-32023-13-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 12/43] refs.c: move resolve_refdup to common","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:47Z","receivedAt":"2015-09-28T22:01:47Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nThis function can be shared across all refs backends so move it\nto the common code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 7 -------\n refs.c | 7 +++++++\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex d46474a..e3f7b1d 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1789,13 +1789,6 @@ const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n \treturn ret;\n }\n \n-char *resolve_refdup(const char *refname, int resolve_flags,\n-\t\t unsigned char *sha1, int *flags)\n-{\n-\treturn xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,\n-\t\t\t\t\t\t sha1, flags));\n-}\n-\n /* The argument to filter_refs */\n struct ref_filter {\n \tconst char *pattern;\ndiff --git a/refs.c b/refs.c\nindex 242f66d..039e5c0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -651,3 +651,10 @@ int ref_exists(const char *refname)\n \tunsigned char sha1[20];\n \treturn !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);\n }\n+\n+char *resolve_refdup(const char *refname, int resolve_flags,\n+\t\t unsigned char *sha1, int *flags)\n+{\n+\treturn xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t\t sha1, flags));\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270850","messageId":"1443477738-32023-14-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 13/43] refs.c: move check_refname_format to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:48Z","receivedAt":"2015-09-28T22:01:48Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nThis function does not contain any backend specific code so we\nmove it to the common code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 109 --------------------------------------------------------\n refs.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 109 insertions(+), 109 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex e3f7b1d..f920b65 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -14,27 +14,6 @@ struct ref_lock {\n };\n \n /*\n- * How to handle various characters in refnames:\n- * 0: An acceptable character for refs\n- * 1: End-of-component\n- * 2: ., look for a preceding . to reject .. in refs\n- * 3: {, look for a preceding @ to reject @{ in refs\n- * 4: A bad character: ASCII control characters, and\n- * \":\", \"?\", \"[\", \"\\\", \"^\", \"~\", SP, or TAB\n- * 5: *, reject unless REFNAME_REFSPEC_PATTERN is set\n- */\n-static unsigned char refname_disposition[256] = {\n-\t1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,\n-\t4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,\n-\t4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 2, 1,\n-\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 4,\n-\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n-\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 4, 0,\n-\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n-\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 4, 4\n-};\n-\n-/*\n * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken\n * refs (i.e., because the reference is about to be deleted anyway).\n */\n@@ -69,94 +48,6 @@ static unsigned char refname_disposition[256] = {\n * value to ref_update::flags\n */\n \n-/*\n- * Try to read one refname component from the front of refname.\n- * Return the length of the component found, or -1 if the component is\n- * not legal. It is legal if it is something reasonable to have under\n- * \".git/refs/\"; We do not like it if:\n- *\n- * - any path component of it begins with \".\", or\n- * - it has double dots \"..\", or\n- * - it has ASCII control characters, or\n- * - it has \":\", \"?\", \"[\", \"\\\", \"^\", \"~\", SP, or TAB anywhere, or\n- * - it has \"*\" anywhere unless REFNAME_REFSPEC_PATTERN is set, or\n- * - it ends with a \"/\", or\n- * - it ends with \".lock\", or\n- * - it contains a \"@{\" portion\n- */\n-static int check_refname_component(const char *refname, int *flags)\n-{\n-\tconst char *cp;\n-\tchar last = '\\0';\n-\n-\tfor (cp = refname; ; cp++) {\n-\t\tint ch = *cp & 255;\n-\t\tunsigned char disp = refname_disposition[ch];\n-\t\tswitch (disp) {\n-\t\tcase 1:\n-\t\t\tgoto out;\n-\t\tcase 2:\n-\t\t\tif (last == '.')\n-\t\t\t\treturn -1; /* Refname contains \"..\". */\n-\t\t\tbreak;\n-\t\tcase 3:\n-\t\t\tif (last == '@')\n-\t\t\t\treturn -1; /* Refname contains \"@{\". */\n-\t\t\tbreak;\n-\t\tcase 4:\n-\t\t\treturn -1;\n-\t\tcase 5:\n-\t\t\tif (!(*flags & REFNAME_REFSPEC_PATTERN))\n-\t\t\t\treturn -1; /* refspec can't be a pattern */\n-\n-\t\t\t/*\n-\t\t\t * Unset the pattern flag so that we only accept\n-\t\t\t * a single asterisk for one side of refspec.\n-\t\t\t */\n-\t\t\t*flags &= ~ REFNAME_REFSPEC_PATTERN;\n-\t\t\tbreak;\n-\t\t}\n-\t\tlast = ch;\n-\t}\n-out:\n-\tif (cp == refname)\n-\t\treturn 0; /* Component has zero length. */\n-\tif (refname[0] == '.')\n-\t\treturn -1; /* Component starts with '.'. */\n-\tif (cp - refname >= LOCK_SUFFIX_LEN &&\n-\t !memcmp(cp - LOCK_SUFFIX_LEN, LOCK_SUFFIX, LOCK_SUFFIX_LEN))\n-\t\treturn -1; /* Refname ends with \".lock\". */\n-\treturn cp - refname;\n-}\n-\n-int check_refname_format(const char *refname, int flags)\n-{\n-\tint component_len, component_count = 0;\n-\n-\tif (!strcmp(refname, \"@\"))\n-\t\t/* Refname is a single character '@'. */\n-\t\treturn -1;\n-\n-\twhile (1) {\n-\t\t/* We are at the start of a path component. */\n-\t\tcomponent_len = check_refname_component(refname, &flags);\n-\t\tif (component_len <= 0)\n-\t\t\treturn -1;\n-\n-\t\tcomponent_count++;\n-\t\tif (refname[component_len] == '\\0')\n-\t\t\tbreak;\n-\t\t/* Skip to next component. */\n-\t\trefname += component_len + 1;\n-\t}\n-\n-\tif (refname[component_len - 1] == '.')\n-\t\treturn -1; /* Refname ends with '.'. */\n-\tif (!(flags & REFNAME_ALLOW_ONELEVEL) && component_count < 2)\n-\t\treturn -1; /* Refname has only one component. */\n-\treturn 0;\n-}\n-\n struct ref_entry;\n \n /*\ndiff --git a/refs.c b/refs.c\nindex 039e5c0..e9cc2d4 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -658,3 +658,112 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \treturn xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,\n \t\t\t\t\t\t sha1, flags));\n }\n+\n+/*\n+ * How to handle various characters in refnames:\n+ * 0: An acceptable character for refs\n+ * 1: End-of-component\n+ * 2: ., look for a preceding . to reject .. in refs\n+ * 3: {, look for a preceding @ to reject @{ in refs\n+ * 4: A bad character: ASCII control characters, and\n+ * \":\", \"?\", \"[\", \"\\\", \"^\", \"~\", SP, or TAB\n+ * 5: *, reject unless REFNAME_REFSPEC_PATTERN is set\n+ */\n+static unsigned char refname_disposition[256] = {\n+\t1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,\n+\t4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,\n+\t4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 2, 1,\n+\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 4,\n+\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 4, 0,\n+\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+\t0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 4, 4\n+};\n+\n+/*\n+ * Try to read one refname component from the front of refname.\n+ * Return the length of the component found, or -1 if the component is\n+ * not legal. It is legal if it is something reasonable to have under\n+ * \".git/refs/\"; We do not like it if:\n+ *\n+ * - any path component of it begins with \".\", or\n+ * - it has double dots \"..\", or\n+ * - it has ASCII control characters, or\n+ * - it has \":\", \"?\", \"[\", \"\\\", \"^\", \"~\", SP, or TAB anywhere, or\n+ * - it has \"*\" anywhere unless REFNAME_REFSPEC_PATTERN is set, or\n+ * - it ends with a \"/\", or\n+ * - it ends with \".lock\", or\n+ * - it contains a \"@{\" portion\n+ */\n+static int check_refname_component(const char *refname, int *flags)\n+{\n+\tconst char *cp;\n+\tchar last = '\\0';\n+\n+\tfor (cp = refname; ; cp++) {\n+\t\tint ch = *cp & 255;\n+\t\tunsigned char disp = refname_disposition[ch];\n+\t\tswitch (disp) {\n+\t\tcase 1:\n+\t\t\tgoto out;\n+\t\tcase 2:\n+\t\t\tif (last == '.')\n+\t\t\t\treturn -1; /* Refname contains \"..\". */\n+\t\t\tbreak;\n+\t\tcase 3:\n+\t\t\tif (last == '@')\n+\t\t\t\treturn -1; /* Refname contains \"@{\". */\n+\t\t\tbreak;\n+\t\tcase 4:\n+\t\t\treturn -1;\n+\t\tcase 5:\n+\t\t\tif (!(*flags & REFNAME_REFSPEC_PATTERN))\n+\t\t\t\treturn -1; /* refspec can't be a pattern */\n+\n+\t\t\t/*\n+\t\t\t * Unset the pattern flag so that we only accept\n+\t\t\t * a single asterisk for one side of refspec.\n+\t\t\t */\n+\t\t\t*flags &= ~ REFNAME_REFSPEC_PATTERN;\n+\t\t\tbreak;\n+\t\t}\n+\t\tlast = ch;\n+\t}\n+out:\n+\tif (cp == refname)\n+\t\treturn 0; /* Component has zero length. */\n+\tif (refname[0] == '.')\n+\t\treturn -1; /* Component starts with '.'. */\n+\tif (cp - refname >= LOCK_SUFFIX_LEN &&\n+\t !memcmp(cp - LOCK_SUFFIX_LEN, LOCK_SUFFIX, LOCK_SUFFIX_LEN))\n+\t\treturn -1; /* Refname ends with \".lock\". */\n+\treturn cp - refname;\n+}\n+\n+int check_refname_format(const char *refname, int flags)\n+{\n+\tint component_len, component_count = 0;\n+\n+\tif (!strcmp(refname, \"@\"))\n+\t\t/* Refname is a single character '@'. */\n+\t\treturn -1;\n+\n+\twhile (1) {\n+\t\t/* We are at the start of a path component. */\n+\t\tcomponent_len = check_refname_component(refname, &flags);\n+\t\tif (component_len <= 0)\n+\t\t\treturn -1;\n+\n+\t\tcomponent_count++;\n+\t\tif (refname[component_len] == '\\0')\n+\t\t\tbreak;\n+\t\t/* Skip to next component. */\n+\t\trefname += component_len + 1;\n+\t}\n+\n+\tif (refname[component_len - 1] == '.')\n+\t\treturn -1; /* Refname ends with '.'. */\n+\tif (!(flags & REFNAME_ALLOW_ONELEVEL) && component_count < 2)\n+\t\treturn -1; /* Refname has only one component. */\n+\treturn 0;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270879","messageId":"1443477738-32023-15-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 14/43] refs.c: move is_branch to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:49Z","receivedAt":"2015-09-28T22:01:49Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 5 -----\n refs.c | 5 +++++\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex f920b65..58ff453 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2969,11 +2969,6 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,\n \treturn ret;\n }\n \n-int is_branch(const char *refname)\n-{\n-\treturn !strcmp(refname, \"HEAD\") || starts_with(refname, \"refs/heads/\");\n-}\n-\n /*\n * Write sha1 into the open lockfile, then close the lockfile. On\n * errors, rollback the lockfile, fill in *err and\ndiff --git a/refs.c b/refs.c\nindex e9cc2d4..bc8750c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -767,3 +767,8 @@ int check_refname_format(const char *refname, int flags)\n \t\treturn -1; /* Refname has only one component. */\n \treturn 0;\n }\n+\n+int is_branch(const char *refname)\n+{\n+\treturn !strcmp(refname, \"HEAD\") || starts_with(refname, \"refs/heads/\");\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270875","messageId":"1443477738-32023-16-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 15/43] refs.c: move prettify_refname to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:50Z","receivedAt":"2015-09-28T22:01:50Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \n---\n refs-be-files.c | 9 ---------\n refs.c | 9 +++++++++\n 2 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 58ff453..9c27851 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2064,15 +2064,6 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-const char *prettify_refname(const char *name)\n-{\n-\treturn name + (\n-\t\tstarts_with(name, \"refs/heads/\") ? 11 :\n-\t\tstarts_with(name, \"refs/tags/\") ? 10 :\n-\t\tstarts_with(name, \"refs/remotes/\") ? 13 :\n-\t\t0);\n-}\n-\n static void unlock_ref(struct ref_lock *lock)\n {\n \t/* Do not free lock->lk -- atexit() still looks at them */\ndiff --git a/refs.c b/refs.c\nindex bc8750c..44ee4f4 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -772,3 +772,12 @@ int is_branch(const char *refname)\n {\n \treturn !strcmp(refname, \"HEAD\") || starts_with(refname, \"refs/heads/\");\n }\n+\n+const char *prettify_refname(const char *name)\n+{\n+\treturn name + (\n+\t\tstarts_with(name, \"refs/heads/\") ? 11 :\n+\t\tstarts_with(name, \"refs/tags/\") ? 10 :\n+\t\tstarts_with(name, \"refs/remotes/\") ? 13 :\n+\t\t0);\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270849","messageId":"1443477738-32023-17-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 16/43] refs.c: move ref iterators to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:51Z","receivedAt":"2015-09-28T22:01:51Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \n---\n refs-be-files.c | 82 ---------------------------------------------------------\n refs.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 81 insertions(+), 82 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 9c27851..5a7e406 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1680,23 +1680,6 @@ const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n \treturn ret;\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n-\tconst char *pattern;\n-\teach_ref_fn *fn;\n-\tvoid *cb_data;\n-};\n-\n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t int flags, void *data)\n-{\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n-\n-\tif (wildmatch(filter->pattern, refname, 0, NULL))\n-\t\treturn 0;\n-\treturn filter->fn(refname, oid, flags, filter->cb_data);\n-}\n-\n enum peel_status {\n \t/* object was peeled successfully: */\n \tPEEL_PEELED = 0,\n@@ -1962,37 +1945,6 @@ int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n {\n \treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);\n }\n-\n-int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in(\"refs/tags/\", fn, cb_data);\n-}\n-\n-int for_each_tag_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in_submodule(submodule, \"refs/tags/\", fn, cb_data);\n-}\n-\n-int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in(\"refs/heads/\", fn, cb_data);\n-}\n-\n-int for_each_branch_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in_submodule(submodule, \"refs/heads/\", fn, cb_data);\n-}\n-\n-int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in(\"refs/remotes/\", fn, cb_data);\n-}\n-\n-int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n-{\n-\treturn for_each_ref_in_submodule(submodule, \"refs/remotes/\", fn, cb_data);\n-}\n-\n int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, git_replace_ref_base, fn,\n@@ -2024,40 +1976,6 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \treturn ret;\n }\n \n-int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n-\tconst char *prefix, void *cb_data)\n-{\n-\tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n-\tint ret;\n-\n-\tif (!prefix && !starts_with(pattern, \"refs/\"))\n-\t\tstrbuf_addstr(&real_pattern, \"refs/\");\n-\telse if (prefix)\n-\t\tstrbuf_addstr(&real_pattern, prefix);\n-\tstrbuf_addstr(&real_pattern, pattern);\n-\n-\tif (!has_glob_specials(pattern)) {\n-\t\t/* Append implied '/' '*' if not present. */\n-\t\tstrbuf_complete(&real_pattern, '/');\n-\t\t/* No need to check for '*', there is none. */\n-\t\tstrbuf_addch(&real_pattern, '*');\n-\t}\n-\n-\tfilter.pattern = real_pattern.buf;\n-\tfilter.fn = fn;\n-\tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n-\n-\tstrbuf_release(&real_pattern);\n-\treturn ret;\n-}\n-\n-int for_each_glob_ref(each_ref_fn fn, const char *pattern, void *cb_data)\n-{\n-\treturn for_each_glob_ref_in(fn, pattern, NULL, cb_data);\n-}\n-\n int for_each_rawref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0,\ndiff --git a/refs.c b/refs.c\nindex 44ee4f4..7714dad 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -781,3 +781,84 @@ const char *prettify_refname(const char *name)\n \t\tstarts_with(name, \"refs/remotes/\") ? 13 :\n \t\t0);\n }\n+\n+/* The argument to filter_refs */\n+struct ref_filter {\n+\tconst char *pattern;\n+\teach_ref_fn *fn;\n+\tvoid *cb_data;\n+};\n+\n+static int filter_refs(const char *refname, const struct object_id *oid, int flags,\n+\t\t void *data)\n+{\n+\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tif (wildmatch(filter->pattern, refname, 0, NULL))\n+\t\treturn 0;\n+\treturn filter->fn(refname, oid, flags, filter->cb_data);\n+}\n+\n+int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n+\tconst char *prefix, void *cb_data)\n+{\n+\tstruct strbuf real_pattern = STRBUF_INIT;\n+\tstruct ref_filter filter;\n+\tint ret;\n+\n+\tif (!prefix && !starts_with(pattern, \"refs/\"))\n+\t\tstrbuf_addstr(&real_pattern, \"refs/\");\n+\telse if (prefix)\n+\t\tstrbuf_addstr(&real_pattern, prefix);\n+\tstrbuf_addstr(&real_pattern, pattern);\n+\n+\tif (!has_glob_specials(pattern)) {\n+\t\t/* Append implied '/' '*' if not present. */\n+\t\tif (real_pattern.buf[real_pattern.len - 1] != '/')\n+\t\t\tstrbuf_addch(&real_pattern, '/');\n+\t\t/* No need to check for '*', there is none. */\n+\t\tstrbuf_addch(&real_pattern, '*');\n+\t}\n+\n+\tfilter.pattern = real_pattern.buf;\n+\tfilter.fn = fn;\n+\tfilter.cb_data = cb_data;\n+\tret = for_each_ref(filter_refs, &filter);\n+\n+\tstrbuf_release(&real_pattern);\n+\treturn ret;\n+}\n+\n+int for_each_glob_ref(each_ref_fn fn, const char *pattern, void *cb_data)\n+{\n+\treturn for_each_glob_ref_in(fn, pattern, NULL, cb_data);\n+}\n+\n+int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in(\"refs/tags/\", fn, cb_data);\n+}\n+\n+int for_each_tag_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in_submodule(submodule, \"refs/tags/\", fn, cb_data);\n+}\n+\n+int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in(\"refs/heads/\", fn, cb_data);\n+}\n+\n+int for_each_branch_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in_submodule(submodule, \"refs/heads/\", fn, cb_data);\n+}\n+\n+int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in(\"refs/remotes/\", fn, cb_data);\n+}\n+\n+int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn for_each_ref_in_submodule(submodule, \"refs/remotes/\", fn, cb_data);\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270876","messageId":"1443477738-32023-18-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 17/43] refs.c: move head_ref_namespaced to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:52Z","receivedAt":"2015-09-28T22:01:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \n---\n refs-be-files.c | 15 ---------------\n refs.c | 15 +++++++++++++++\n 2 files changed, 15 insertions(+), 15 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 5a7e406..05a4b88 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1951,21 +1951,6 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n \t\t\t strlen(git_replace_ref_base), 0, cb_data);\n }\n \n-int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tint ret = 0;\n-\tstruct object_id oid;\n-\tint flag;\n-\n-\tstrbuf_addf(&buf, \"%sHEAD\", get_git_namespace());\n-\tif (!read_ref_full(buf.buf, RESOLVE_REF_READING, oid.hash, &flag))\n-\t\tret = fn(buf.buf, &oid, flag, cb_data);\n-\tstrbuf_release(&buf);\n-\n-\treturn ret;\n-}\n-\n int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/refs.c b/refs.c\nindex 7714dad..0f4e19a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -862,3 +862,18 @@ int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *c\n {\n \treturn for_each_ref_in_submodule(submodule, \"refs/remotes/\", fn, cb_data);\n }\n+\n+int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret = 0;\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tstrbuf_addf(&buf, \"%sHEAD\", get_git_namespace());\n+\tif (!read_ref_full(buf.buf, RESOLVE_REF_READING, oid.hash, &flag))\n+\t\tret = fn(buf.buf, &oid, flag, cb_data);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270851","messageId":"1443477738-32023-19-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 18/43] refs-be-files.c: add a backend method structure with transaction functions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:53Z","receivedAt":"2015-09-28T22:01:53Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd a ref structure for backend methods. Start by adding method pointers\nfor the transaction functions.\n\nAdd a function set_refs_backend to switch between backends. The files\nbased backend is the default.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 62 +++++++++++++++++++++++++++-------------------\n refs.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n refs.h | 36 +++++++++++++++++++++++++++\n 3 files changed, 150 insertions(+), 25 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 05a4b88..6e6f092 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3310,14 +3310,14 @@ struct ref_transaction {\n \tenum ref_transaction_state state;\n };\n \n-struct ref_transaction *ref_transaction_begin(struct strbuf *err)\n+static struct ref_transaction *files_transaction_begin(struct strbuf *err)\n {\n \tassert(err);\n \n \treturn xcalloc(1, sizeof(struct ref_transaction));\n }\n \n-void ref_transaction_free(struct ref_transaction *transaction)\n+static void files_transaction_free(struct ref_transaction *transaction)\n {\n \tint i;\n \n@@ -3344,12 +3344,12 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \treturn update;\n }\n \n-int ref_transaction_update(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *new_sha1,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t unsigned int flags, const char *msg,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_update(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *new_sha1,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n {\n \tstruct ref_update *update;\n \n@@ -3380,11 +3380,11 @@ int ref_transaction_update(struct ref_transaction *transaction,\n \treturn 0;\n }\n \n-int ref_transaction_create(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *new_sha1,\n-\t\t\t unsigned int flags, const char *msg,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_create(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *new_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n {\n \tif (!new_sha1 || is_null_sha1(new_sha1))\n \t\tdie(\"BUG: create called without valid new_sha1\");\n@@ -3392,11 +3392,11 @@ int ref_transaction_create(struct ref_transaction *transaction,\n \t\t\t\t null_sha1, flags, msg, err);\n }\n \n-int ref_transaction_delete(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t unsigned int flags, const char *msg,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_delete(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n {\n \tif (old_sha1 && is_null_sha1(old_sha1))\n \t\tdie(\"BUG: delete called with old_sha1 set to zeros\");\n@@ -3405,11 +3405,11 @@ int ref_transaction_delete(struct ref_transaction *transaction,\n \t\t\t\t flags, msg, err);\n }\n \n-int ref_transaction_verify(struct ref_transaction *transaction,\n-\t\t\t const char *refname,\n-\t\t\t const unsigned char *old_sha1,\n-\t\t\t unsigned int flags,\n-\t\t\t struct strbuf *err)\n+int files_transaction_verify(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t struct strbuf *err)\n {\n \tif (!old_sha1)\n \t\tdie(\"BUG: verify called with old_sha1 set to NULL\");\n@@ -3435,8 +3435,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-int ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t struct strbuf *err)\n+static int files_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3822,3 +3822,15 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \tunlock_ref(lock);\n \treturn -1;\n }\n+\n+struct ref_be refs_be_files = {\n+\tNULL,\n+\t\"files\",\n+\tfiles_transaction_begin,\n+\tfiles_transaction_update,\n+\tfiles_transaction_create,\n+\tfiles_transaction_delete,\n+\tfiles_transaction_verify,\n+\tfiles_transaction_commit,\n+\tfiles_transaction_free,\n+};\ndiff --git a/refs.c b/refs.c\nindex 0f4e19a..98aa357 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -4,6 +4,29 @@\n #include \"cache.h\"\n #include \"refs.h\"\n #include \"lockfile.h\"\n+/*\n+ * We always have a files backend and it is the default.\n+ */\n+struct ref_be *the_refs_backend = &refs_be_files;\n+/*\n+ * List of all available backends\n+ */\n+struct ref_be *refs_backends = &refs_be_files;\n+\n+/*\n+ * This function is used to switch to an alternate backend.\n+ */\n+int set_refs_backend(const char *name)\n+{\n+\tstruct ref_be *be;\n+\n+\tfor (be = refs_backends; be; be = be->next)\n+\t\tif (!strcmp(be->name, name)) {\n+\t\t\tthe_refs_backend = be;\n+\t\t\treturn 0;\n+\t\t}\n+\treturn 1;\n+}\n \n static int is_per_worktree_ref(const char *refname)\n {\n@@ -877,3 +900,57 @@ int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n \n \treturn ret;\n }\n+\n+\n+/* backend functions */\n+struct ref_transaction *ref_transaction_begin(struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_begin(err);\n+}\n+\n+\n+int ref_transaction_update(struct ref_transaction *transaction,\n+\t\t\t const char *refname, const unsigned char *new_sha1,\n+\t\t\t const unsigned char *old_sha1, unsigned int flags,\n+\t\t\t const char *msg, struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_update(transaction,\n+\t\t\trefname, new_sha1, old_sha1, flags, msg, err);\n+}\n+\n+int ref_transaction_create(struct ref_transaction *transaction,\n+\t\t\t const char *refname, const unsigned char *new_sha1,\n+\t\t\t unsigned int flags, const char *msg,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_create(transaction,\n+\t\t\trefname, new_sha1, flags, msg, err);\n+}\n+int ref_transaction_delete(struct ref_transaction *transaction,\n+\t\t\t const char *refname, const unsigned char *old_sha1,\n+\t\t\t unsigned int flags, const char *msg,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_delete(transaction,\n+\t\t\trefname, old_sha1, flags, msg, err);\n+}\n+\n+int ref_transaction_verify(struct ref_transaction *transaction,\n+\t\t\t const char *refname, const unsigned char *old_sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_verify(transaction,\n+\t\t\trefname, old_sha1, flags, err);\n+}\n+\n+int ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->transaction_commit(transaction, err);\n+}\n+\n+void ref_transaction_free(struct ref_transaction *transaction)\n+{\n+\treturn the_refs_backend->transaction_free(transaction);\n+}\ndiff --git a/refs.h b/refs.h\nindex 729bc3c..a1db3ef 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -530,4 +530,40 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n+/* refs backends */\n+typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n+typedef int (*ref_transaction_update_fn)(struct ref_transaction *transaction,\n+\t\tconst char *refname, const unsigned char *new_sha1,\n+\t\tconst unsigned char *old_sha1, unsigned int flags,\n+\t\tconst char *msg, struct strbuf *err);\n+typedef int (*ref_transaction_create_fn)(\n+\t\tstruct ref_transaction *transaction,\n+\t\tconst char *refname, const unsigned char *new_sha1,\n+\t\tunsigned int flags, const char *msg, struct strbuf *err);\n+typedef int (*ref_transaction_delete_fn)(struct ref_transaction *transaction,\n+\t\tconst char *refname, const unsigned char *old_sha1,\n+\t\tunsigned int flags, const char *msg, struct strbuf *err);\n+typedef int (*ref_transaction_verify_fn)(struct ref_transaction *transaction,\n+\t\tconst char *refname, const unsigned char *old_sha1,\n+\t\tunsigned int flags, struct strbuf *err);\n+typedef int (*ref_transaction_commit_fn)(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err);\n+typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n+\n+struct ref_be {\n+\tstruct ref_be *next;\n+\tconst char *name;\n+\tref_transaction_begin_fn transaction_begin;\n+\tref_transaction_update_fn transaction_update;\n+\tref_transaction_create_fn transaction_create;\n+\tref_transaction_delete_fn transaction_delete;\n+\tref_transaction_verify_fn transaction_verify;\n+\tref_transaction_commit_fn transaction_commit;\n+\tref_transaction_free_fn transaction_free;\n+};\n+\n+\n+extern struct ref_be refs_be_files;\n+int set_refs_backend(const char *name);\n+\n #endif /* REFS_H */\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270877","messageId":"1443477738-32023-20-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 19/43] refs-be-files.c: add methods for misc ref operations","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:54Z","receivedAt":"2015-09-28T22:01:54Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nAdd ref backend methods for:\nresolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,\ncreate_symref, resolve_gitlink_ref.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n cache.h | 7 +++++++\n refs-be-files.c | 31 ++++++++++++++++++++++---------\n refs.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n refs.h | 21 +++++++++++++++++++++\n 4 files changed, 98 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 497d296..1c45b62 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1104,6 +1104,13 @@ extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as s\n extern int interpret_branch_name(const char *str, int len, struct strbuf *);\n extern int get_sha1_mb(const char *str, unsigned char *sha1);\n \n+/*\n+ * Return true iff abbrev_name is a possible abbreviation for\n+ * full_name according to the rules defined by ref_rev_parse_rules in\n+ * refs.c.\n+ */\n+extern int refname_match(const char *abbrev_name, const char *full_name);\n+\n extern int validate_headref(const char *ref);\n \n extern int base_name_compare(const char *name1, int len1, int mode1, const char *name2, int len2, int mode2);\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 6e6f092..7101e31 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1433,7 +1433,8 @@ static int resolve_gitlink_ref_recursive(struct ref_cache *refs,\n \treturn resolve_gitlink_ref_recursive(refs, p, sha1, recursion+1);\n }\n \n-int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *sha1)\n+static int files_resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n {\n \tint len = strlen(path), retval;\n \tchar *submodule;\n@@ -1665,8 +1666,10 @@ static const char *resolve_ref_1(const char *refname,\n \t}\n }\n \n-const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n-\t\t\t unsigned char *sha1, int *flags)\n+static const char *files_resolve_ref_unsafe(const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n {\n \tstatic struct strbuf sb_refname = STRBUF_INIT;\n \tstruct strbuf sb_contents = STRBUF_INIT;\n@@ -1768,7 +1771,7 @@ static enum peel_status peel_entry(struct ref_entry *entry, int repeel)\n \treturn status;\n }\n \n-int peel_ref(const char *refname, unsigned char *sha1)\n+static int files_peel_ref(const char *refname, unsigned char *sha1)\n {\n \tint flag;\n \tunsigned char base[20];\n@@ -2402,7 +2405,7 @@ static void prune_refs(struct ref_to_prune *r)\n \t}\n }\n \n-int pack_refs(unsigned int flags)\n+static int files_pack_refs(unsigned int flags)\n {\n \tstruct pack_refs_cb_data cbdata;\n \n@@ -2724,8 +2727,10 @@ static int should_autocreate_reflog(const char *refname)\n \t\t!strcmp(refname, \"HEAD\");\n }\n \n-int verify_refname_available(const char *newname, struct string_list *extra,\n-\t\t\t struct string_list *skip, struct strbuf *err)\n+static int files_verify_refname_available(const char *newname,\n+\t\t\t\t\t struct string_list *extra,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tstruct ref_dir *packed_refs = get_packed_refs(&ref_cache);\n \tstruct ref_dir *loose_refs = get_loose_refs(&ref_cache);\n@@ -2960,8 +2965,10 @@ static int commit_ref_update(struct ref_lock *lock,\n \treturn 0;\n }\n \n-int create_symref(const char *ref_target, const char *refs_heads_master,\n-\t\t const char *logmsg)\n+static int files_create_symref(struct ref_transaction *trans,\n+\t\t\t const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg)\n {\n \tchar *lockpath = NULL;\n \tchar ref[1000];\n@@ -3833,4 +3840,10 @@ struct ref_be refs_be_files = {\n \tfiles_transaction_verify,\n \tfiles_transaction_commit,\n \tfiles_transaction_free,\n+\tfiles_resolve_ref_unsafe,\n+\tfiles_verify_refname_available,\n+\tfiles_pack_refs,\n+\tfiles_peel_ref,\n+\tfiles_create_symref,\n+\tfiles_resolve_gitlink_ref,\n };\ndiff --git a/refs.c b/refs.c\nindex 98aa357..a9901f3 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -954,3 +954,51 @@ void ref_transaction_free(struct ref_transaction *transaction)\n {\n \treturn the_refs_backend->transaction_free(transaction);\n }\n+\n+const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n+\t\t\t unsigned char *sha1, int *flags)\n+{\n+\treturn the_refs_backend->resolve_ref_unsafe(ref, resolve_flags, sha1,\n+\t\t\t\t\t\t flags);\n+}\n+\n+int verify_refname_available(const char *refname, struct string_list *extra,\n+\t\t\t struct string_list *skip, struct strbuf *err)\n+{\n+\treturn the_refs_backend->verify_refname_available(refname, extra, skip,\n+\t\t\t\t\t\t\t err);\n+}\n+\n+int pack_refs(unsigned int flags)\n+{\n+\treturn the_refs_backend->pack_refs(flags);\n+}\n+\n+int peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\treturn the_refs_backend->peel_ref(refname, sha1);\n+}\n+\n+int create_symref(const char *ref_target, const char *refs_heads_master,\n+\t\t const char *logmsg)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction = ref_transaction_begin(&err);\n+\tif (!transaction)\n+\t\treturn -1;\n+\n+\tif (the_refs_backend->create_symref(transaction, ref_target,\n+\t\t\t\t\t refs_heads_master, logmsg))\n+\t\treturn -1;\n+\n+\tif (ref_transaction_commit(transaction, &err))\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+int resolve_gitlink_ref(const char *path, const char *refname,\n+\t\t\tunsigned char *sha1)\n+{\n+\treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n+}\ndiff --git a/refs.h b/refs.h\nindex a1db3ef..6e88909 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -549,6 +549,21 @@ typedef int (*ref_transaction_verify_fn)(struct ref_transaction *transaction,\n typedef int (*ref_transaction_commit_fn)(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n+typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags);\n+typedef int (*verify_refname_available_fn)(const char *refname,\n+\t\t\t\t\t struct string_list *extra,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err);\n+typedef int (*pack_refs_fn)(unsigned int flags);\n+typedef int (*peel_ref_fn)(const char *refname, unsigned char *sha1);\n+typedef int (*create_symref_fn)(struct ref_transaction *transaction,\n+\t\t\t\tconst char *ref_target,\n+\t\t\t\tconst char *refs_heads_master,\n+\t\t\t\tconst char *logmsg);\n+typedef int (*resolve_gitlink_ref_fn)(const char *path, const char *refname,\n+\t\t\t\t unsigned char *sha1);\n \n struct ref_be {\n \tstruct ref_be *next;\n@@ -560,6 +575,12 @@ struct ref_be {\n \tref_transaction_verify_fn transaction_verify;\n \tref_transaction_commit_fn transaction_commit;\n \tref_transaction_free_fn transaction_free;\n+\tresolve_ref_unsafe_fn resolve_ref_unsafe;\n+\tverify_refname_available_fn verify_refname_available;\n+\tpack_refs_fn pack_refs;\n+\tpeel_ref_fn peel_ref;\n+\tcreate_symref_fn create_symref;\n+\tresolve_gitlink_ref_fn resolve_gitlink_ref;\n };\n \n \n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270848","messageId":"1443477738-32023-21-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 20/43] refs-be-files.c: add methods for the ref iterators","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:55Z","receivedAt":"2015-09-28T22:01:55Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Ronnie Sahlberg \n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 41 +++++++++++++++++++++++++++++------------\n refs.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n refs.h | 26 ++++++++++++++++++++++++++\n 3 files changed, 109 insertions(+), 12 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 7101e31..9b45468 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1909,32 +1909,36 @@ static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n \treturn 0;\n }\n \n-int head_ref(each_ref_fn fn, void *cb_data)\n+static int files_head_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_head_ref(NULL, fn, cb_data);\n }\n \n-int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_head_ref(submodule, fn, cb_data);\n }\n \n-int for_each_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t\tvoid *cb_data)\n {\n \treturn do_for_each_ref(get_ref_cache(submodule), \"\", fn, 0, 0, cb_data);\n }\n \n-int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, prefix, fn, strlen(prefix), 0, cb_data);\n }\n \n-int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)\n+static int files_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n {\n \tunsigned int flag = 0;\n \n@@ -1943,18 +1947,21 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig\n \treturn do_for_each_ref(&ref_cache, prefix, fn, 0, flag, cb_data);\n }\n \n-int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n-\t\teach_ref_fn fn, void *cb_data)\n+static int files_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(get_ref_cache(submodule), prefix, fn,\n+\t\t\t strlen(prefix), 0, cb_data);\n }\n-int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+\n+static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, git_replace_ref_base, fn,\n \t\t\t strlen(git_replace_ref_base), 0, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n@@ -1964,7 +1971,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \treturn ret;\n }\n \n-int for_each_rawref(each_ref_fn fn, void *cb_data)\n+static int files_for_each_rawref(each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(&ref_cache, \"\", fn, 0,\n \t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n@@ -3846,4 +3853,14 @@ struct ref_be refs_be_files = {\n \tfiles_peel_ref,\n \tfiles_create_symref,\n \tfiles_resolve_gitlink_ref,\n+\tfiles_head_ref,\n+\tfiles_head_ref_submodule,\n+\tfiles_for_each_ref,\n+\tfiles_for_each_ref_submodule,\n+\tfiles_for_each_ref_in,\n+\tfiles_for_each_fullref_in,\n+\tfiles_for_each_ref_in_submodule,\n+\tfiles_for_each_rawref,\n+\tfiles_for_each_namespaced_ref,\n+\tfiles_for_each_replace_ref,\n };\ndiff --git a/refs.c b/refs.c\nindex a9901f3..0375bb5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1002,3 +1002,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,\n {\n \treturn the_refs_backend->resolve_gitlink_ref(path, refname, sha1);\n }\n+\n+int head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref(fn, cb_data);\n+}\n+\n+int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->head_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref(fn, cb_data);\n+}\n+\n+int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_submodule(submodule, fn, cb_data);\n+}\n+\n+int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in(prefix, fn, cb_data);\n+}\n+\n+int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data,\n+\t\t\tunsigned int broken)\n+{\n+\treturn the_refs_backend->for_each_fullref_in(prefix, fn, cb_data,\n+\t\t\t\t\t\t broken);\n+}\n+\n+int for_each_ref_in_submodule(const char *submodule, const char *prefix,\n+\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_ref_in_submodule(submodule, prefix,\n+\t\t\t\t\t\t\t fn, cb_data);\n+}\n+\n+int for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_rawref(fn, cb_data);\n+}\n+\n+int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_namespaced_ref(fn, cb_data);\n+}\n+\n+int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n+}\ndiff --git a/refs.h b/refs.h\nindex 6e88909..b26cffc 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -564,6 +564,22 @@ typedef int (*create_symref_fn)(struct ref_transaction *transaction,\n \t\t\t\tconst char *logmsg);\n typedef int (*resolve_gitlink_ref_fn)(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n+typedef int (*head_ref_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*head_ref_submodule_fn)(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int (*for_each_ref_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*for_each_ref_submodule_fn)(const char *submodule, each_ref_fn fn,\n+\t\t\t\t\t void *cb_data);\n+typedef int (*for_each_ref_in_fn)(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int (*for_each_fullref_in_fn)(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken);\n+typedef int (*for_each_ref_in_submodule_fn)(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data);\n+typedef int (*for_each_rawref_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*for_each_namespaced_ref_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*for_each_replace_ref_fn)(each_ref_fn fn, void *cb_data);\n \n struct ref_be {\n \tstruct ref_be *next;\n@@ -581,6 +597,16 @@ struct ref_be {\n \tpeel_ref_fn peel_ref;\n \tcreate_symref_fn create_symref;\n \tresolve_gitlink_ref_fn resolve_gitlink_ref;\n+\thead_ref_fn head_ref;\n+\thead_ref_submodule_fn head_ref_submodule;\n+\tfor_each_ref_fn for_each_ref;\n+\tfor_each_ref_submodule_fn for_each_ref_submodule;\n+\tfor_each_ref_in_fn for_each_ref_in;\n+\tfor_each_fullref_in_fn for_each_fullref_in;\n+\tfor_each_ref_in_submodule_fn for_each_ref_in_submodule;\n+\tfor_each_rawref_fn for_each_rawref;\n+\tfor_each_namespaced_ref_fn for_each_namespaced_ref;\n+\tfor_each_replace_ref_fn for_each_replace_ref;\n };\n \n \n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270871","messageId":"1443477738-32023-22-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 21/43] refs-be-files.c: add method for for_each_reftype_...","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:56Z","receivedAt":"2015-09-28T22:01:56Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add method for for_each_reftype_fullpath.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 11 +++++++++++\n refs.c | 7 +++++++\n refs.h | 3 +++\n 3 files changed, 21 insertions(+)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 9b45468..eb18a20 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1961,6 +1961,16 @@ static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n \t\t\t strlen(git_replace_ref_base), 0, cb_data);\n }\n \n+static int files_for_each_reftype_fullpath(each_ref_fn fn, char *type,\n+\t\t\t\t\t unsigned int broken, void *cb_data)\n+{\n+\tunsigned int flag = 0;\n+\n+\tif (broken)\n+\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n+\treturn do_for_each_ref(&ref_cache, type, fn, 0, flag, cb_data);\n+}\n+\n static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -3863,4 +3873,5 @@ struct ref_be refs_be_files = {\n \tfiles_for_each_rawref,\n \tfiles_for_each_namespaced_ref,\n \tfiles_for_each_replace_ref,\n+\tfiles_for_each_reftype_fullpath,\n };\ndiff --git a/refs.c b/refs.c\nindex 0375bb5..c6112ac 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1056,3 +1056,10 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)\n {\n \treturn the_refs_backend->for_each_replace_ref(fn, cb_data);\n }\n+\n+int for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken,\n+\t\t\t void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reftype_fullpath(fn, type, broken,\n+\t\t\t\t\t\t\t cb_data);\n+}\ndiff --git a/refs.h b/refs.h\nindex b26cffc..5875fe5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -580,6 +580,8 @@ typedef int (*for_each_ref_in_submodule_fn)(const char *submodule,\n typedef int (*for_each_rawref_fn)(each_ref_fn fn, void *cb_data);\n typedef int (*for_each_namespaced_ref_fn)(each_ref_fn fn, void *cb_data);\n typedef int (*for_each_replace_ref_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*for_each_reftype_fullpath_fn)(each_ref_fn fn, char *type,\n+\t\t\t\t\t unsigned int broken, void *cb_data);\n \n struct ref_be {\n \tstruct ref_be *next;\n@@ -607,6 +609,7 @@ struct ref_be {\n \tfor_each_rawref_fn for_each_rawref;\n \tfor_each_namespaced_ref_fn for_each_namespaced_ref;\n \tfor_each_replace_ref_fn for_each_replace_ref;\n+\tfor_each_reftype_fullpath_fn for_each_reftype_fullpath;\n };\n \n \n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270873","messageId":"1443477738-32023-23-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 22/43] refs-be-files.c: add do_for_each_per_worktree_ref","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:57Z","receivedAt":"2015-09-28T22:01:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might still use files to store per-worktree\nrefs. So the files backend's ref-loading infrastructure should be\navailable to those backends, just for use on per-worktree refs. Add\ndo_for_each_per_worktree_ref, which iterates over per-worktree refs.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 15 ++++++++++++---\n refs.h | 11 +++++++++++\n 2 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex eb18a20..dc89289 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -587,9 +587,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n \tdir->sorted = dir->nr = i;\n }\n \n-/* Include broken references in a do_for_each_ref*() iteration: */\n-#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n-\n /*\n * Return true iff the reference described by entry can be resolved to\n * an object in the database. Emit a warning if the referred-to\n@@ -637,6 +634,10 @@ static int do_one_ref(struct ref_entry *entry, void *cb_data)\n \tstruct ref_entry *old_current_ref;\n \tint retval;\n \n+\tif (data->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t ref_type(entry->name) != REF_TYPE_PER_WORKTREE)\n+\t\treturn 0;\n+\n \tif (!starts_with(entry->name, data->base))\n \t\treturn 0;\n \n@@ -1891,6 +1892,14 @@ static int do_for_each_ref(struct ref_cache *refs, const char *base,\n \treturn do_for_each_entry(refs, base, do_one_ref, &data);\n }\n \n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(get_ref_cache(submodule), base, fn, trim,\n+\t\t\t flags | DO_FOR_EACH_PER_WORKTREE_ONLY, cb_data);\n+}\n+\n static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n {\n \tstruct object_id oid;\ndiff --git a/refs.h b/refs.h\nindex 5875fe5..09d140d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -152,6 +152,12 @@ struct ref_transaction;\n */\n #define REF_BAD_NAME 0x08\n \n+/* Include broken references in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n+\n+/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n+#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n+\n /*\n * The signature for the callback function for the for_each_*()\n * functions below. The memory pointed to by the refname and sha1\n@@ -195,6 +201,11 @@ extern int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n /* can be used to learn about broken ref and symref */\n extern int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+\n+int do_for_each_per_worktree_ref(const char *submodule, const char *base,\n+\t\t\t\t each_ref_fn fn, int trim, int flags,\n+\t\t\t\t void *cb_data);\n+\n static inline const char *has_glob_specials(const char *pattern)\n {\n \treturn strpbrk(pattern, \"?*[\");\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270872","messageId":"1443477738-32023-24-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 23/43] refs.c: move refname_is_safe to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:58Z","receivedAt":"2015-09-28T22:01:58Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This function does not contain any backend specific code so we\nmove it to the common code.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 33 ---------------------------------\n refs.c | 23 +++++++++++++++++++++++\n refs.h | 2 ++\n 3 files changed, 25 insertions(+), 33 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex dc89289..bb5a3ad 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -231,39 +231,6 @@ static struct ref_dir *get_ref_dir(struct ref_entry *entry)\n \treturn dir;\n }\n \n-/*\n- * Check if a refname is safe.\n- * For refs that start with \"refs/\" we consider it safe as long they do\n- * not try to resolve to outside of refs/.\n- *\n- * For all other refs we only consider them safe iff they only contain\n- * upper case characters and '_' (like \"HEAD\" AND \"MERGE_HEAD\", and not like\n- * \"config\").\n- */\n-static int refname_is_safe(const char *refname)\n-{\n-\tif (starts_with(refname, \"refs/\")) {\n-\t\tchar *buf;\n-\t\tint result;\n-\n-\t\tbuf = xmalloc(strlen(refname) + 1);\n-\t\t/*\n-\t\t * Does the refname try to escape refs/?\n-\t\t * For example: refs/foo/../bar is safe but refs/foo/../../bar\n-\t\t * is not.\n-\t\t */\n-\t\tresult = !normalize_path_copy(buf, refname + strlen(\"refs/\"));\n-\t\tfree(buf);\n-\t\treturn result;\n-\t}\n-\twhile (*refname) {\n-\t\tif (!isupper(*refname) && *refname != '_')\n-\t\t\treturn 0;\n-\t\trefname++;\n-\t}\n-\treturn 1;\n-}\n-\n static struct ref_entry *create_ref_entry(const char *refname,\n \t\t\t\t\t const unsigned char *sha1, int flag,\n \t\t\t\t\t int check_name)\ndiff --git a/refs.c b/refs.c\nindex c6112ac..aa49db5 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -901,6 +901,29 @@ int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n \treturn ret;\n }\n \n+int refname_is_safe(const char *refname)\n+{\n+\tif (starts_with(refname, \"refs/\")) {\n+\t\tchar *buf;\n+\t\tint result;\n+\n+\t\tbuf = xmalloc(strlen(refname) + 1);\n+\t\t/*\n+\t\t * Does the refname try to escape refs/?\n+\t\t * For example: refs/foo/../bar is safe but refs/foo/../../bar\n+\t\t * is not.\n+\t\t */\n+\t\tresult = !normalize_path_copy(buf, refname + strlen(\"refs/\"));\n+\t\tfree(buf);\n+\t\treturn result;\n+\t}\n+\twhile (*refname) {\n+\t\tif (!isupper(*refname) && *refname != '_')\n+\t\t\treturn 0;\n+\t\trefname++;\n+\t}\n+\treturn 1;\n+}\n \n /* backend functions */\n struct ref_transaction *ref_transaction_begin(struct strbuf *err)\ndiff --git a/refs.h b/refs.h\nindex 09d140d..2bafa55 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -326,6 +326,8 @@ extern int for_each_reflog(each_ref_fn, void *);\n */\n extern int check_refname_format(const char *refname, int flags);\n \n+extern int refname_is_safe(const char *refname);\n+\n extern const char *prettify_refname(const char *refname);\n \n extern char *shorten_unambiguous_ref(const char *refname, int strict);\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270874","messageId":"1443477738-32023-25-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 24/43] refs.h: document make refname_is_safe and add it to header","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:01:59Z","receivedAt":"2015-09-28T22:01:59Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This function might be used by other refs backends\n\nSigned-off-by: David Turner \n---\n refs.h | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/refs.h b/refs.h\nindex 2bafa55..db60bfe 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -259,6 +259,17 @@ int verify_refname_available(const char *newname, struct string_list *extra,\n \t\t\t struct string_list *skip, struct strbuf *err);\n \n /*\n+ * Check if a refname is safe.\n+ * For refs that start with \"refs/\" we consider it safe as long they do\n+ * not try to resolve to outside of refs/.\n+ *\n+ * For all other refs we only consider them safe iff they only contain\n+ * upper case characters and '_' (like \"HEAD\" AND \"MERGE_HEAD\", and not like\n+ * \"config\").\n+ */\n+int refname_is_safe(const char *refname);\n+\n+/*\n * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.\n * REF_NODEREF: act on the ref directly, instead of dereferencing\n * symbolic references.\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270852","messageId":"1443477738-32023-26-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 25/43] refs.c: move copy_msg to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:00Z","receivedAt":"2015-09-28T22:02:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Rename copy_msg to copy_reflog_msg and make it public.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 28 +---------------------------\n refs.c | 26 ++++++++++++++++++++++++++\n refs.h | 2 ++\n 3 files changed, 29 insertions(+), 27 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex bb5a3ad..8898e86 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2684,32 +2684,6 @@ static int commit_ref(struct ref_lock *lock)\n \treturn 0;\n }\n \n-/*\n- * copy the reflog message msg to buf, which has been allocated sufficiently\n- * large, while cleaning up the whitespaces. Especially, convert LF to space,\n- * because reflog file is one line per entry.\n- */\n-static int copy_msg(char *buf, const char *msg)\n-{\n-\tchar *cp = buf;\n-\tchar c;\n-\tint wasspace = 1;\n-\n-\t*cp++ = '\\t';\n-\twhile ((c = *msg++)) {\n-\t\tif (wasspace && isspace(c))\n-\t\t\tcontinue;\n-\t\twasspace = isspace(c);\n-\t\tif (wasspace)\n-\t\t\tc = ' ';\n-\t\t*cp++ = c;\n-\t}\n-\twhile (buf < cp && isspace(cp[-1]))\n-\t\tcp--;\n-\t*cp++ = '\\n';\n-\treturn cp - buf;\n-}\n-\n static int should_autocreate_reflog(const char *refname)\n {\n \tif (!log_all_ref_updates)\n@@ -2806,7 +2780,7 @@ static int log_ref_write_fd(int fd, const unsigned char *old_sha1,\n \t\t\tsha1_to_hex(new_sha1),\n \t\t\tcommitter);\n \tif (msglen)\n-\t\tlen += copy_msg(logrec + len - 1, msg) - 1;\n+\t\tlen += copy_reflog_msg(logrec + len - 1, msg) - 1;\n \n \twritten = len <= maxlen ? write_in_full(fd, logrec, len) : -1;\n \tfree(logrec);\ndiff --git a/refs.c b/refs.c\nindex aa49db5..db650b6 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -886,6 +886,32 @@ int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *c\n \treturn for_each_ref_in_submodule(submodule, \"refs/remotes/\", fn, cb_data);\n }\n \n+/*\n+ * copy the reflog message msg to buf, which has been allocated sufficiently\n+ * large, while cleaning up the whitespaces. Especially, convert LF to space,\n+ * because reflog file is one line per entry.\n+ */\n+int copy_reflog_msg(char *buf, const char *msg)\n+{\n+\tchar *cp = buf;\n+\tchar c;\n+\tint wasspace = 1;\n+\n+\t*cp++ = '\\t';\n+\twhile ((c = *msg++)) {\n+\t\tif (wasspace && isspace(c))\n+\t\t\tcontinue;\n+\t\twasspace = isspace(c);\n+\t\tif (wasspace)\n+\t\t\tc = ' ';\n+\t\t*cp++ = c;\n+\t}\n+\twhile (buf < cp && isspace(cp[-1]))\n+\t\tcp--;\n+\t*cp++ = '\\n';\n+\treturn cp - buf;\n+}\n+\n int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/refs.h b/refs.h\nindex db60bfe..c224f0e 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -508,6 +508,8 @@ enum ref_type {\n \n enum ref_type ref_type(const char *refname);\n \n+int copy_reflog_msg(char *buf, const char *msg);\n+\n enum expire_reflog_flags {\n \tEXPIRE_REFLOGS_DRY_RUN = 1 << 0,\n \tEXPIRE_REFLOGS_UPDATE_REF = 1 << 1,\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270870","messageId":"1443477738-32023-27-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 26/43] refs.c: move peel_object to the common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:01Z","receivedAt":"2015-09-28T22:02:01Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This function does not contain any backend specific code so we\nmove it to the common code.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 53 -----------------------------------------------------\n refs.c | 31 +++++++++++++++++++++++++++++++\n refs.h | 27 +++++++++++++++++++++++++++\n 3 files changed, 58 insertions(+), 53 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 8898e86..5971526 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -1651,59 +1651,6 @@ static const char *files_resolve_ref_unsafe(const char *refname,\n \treturn ret;\n }\n \n-enum peel_status {\n-\t/* object was peeled successfully: */\n-\tPEEL_PEELED = 0,\n-\n-\t/*\n-\t * object cannot be peeled because the named object (or an\n-\t * object referred to by a tag in the peel chain), does not\n-\t * exist.\n-\t */\n-\tPEEL_INVALID = -1,\n-\n-\t/* object cannot be peeled because it is not a tag: */\n-\tPEEL_NON_TAG = -2,\n-\n-\t/* ref_entry contains no peeled value because it is a symref: */\n-\tPEEL_IS_SYMREF = -3,\n-\n-\t/*\n-\t * ref_entry cannot be peeled because it is broken (i.e., the\n-\t * symbolic reference cannot even be resolved to an object\n-\t * name):\n-\t */\n-\tPEEL_BROKEN = -4\n-};\n-\n-/*\n- * Peel the named object; i.e., if the object is a tag, resolve the\n- * tag recursively until a non-tag is found. If successful, store the\n- * result to sha1 and return PEEL_PEELED. If the object is not a tag\n- * or is not valid, return PEEL_NON_TAG or PEEL_INVALID, respectively,\n- * and leave sha1 unchanged.\n- */\n-static enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)\n-{\n-\tstruct object *o = lookup_unknown_object(name);\n-\n-\tif (o->type == OBJ_NONE) {\n-\t\tint type = sha1_object_info(name, NULL);\n-\t\tif (type < 0 || !object_as_type(o, type, 0))\n-\t\t\treturn PEEL_INVALID;\n-\t}\n-\n-\tif (o->type != OBJ_TAG)\n-\t\treturn PEEL_NON_TAG;\n-\n-\to = deref_tag_noverify(o);\n-\tif (!o)\n-\t\treturn PEEL_INVALID;\n-\n-\thashcpy(sha1, o->sha1);\n-\treturn PEEL_PEELED;\n-}\n-\n /*\n * Peel the entry (if possible) and return its new peel_status. If\n * repeel is true, re-peel the entry even if there is an old peeled\ndiff --git a/refs.c b/refs.c\nindex db650b6..d13b854 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -4,6 +4,9 @@\n #include \"cache.h\"\n #include \"refs.h\"\n #include \"lockfile.h\"\n+#include \"object.h\"\n+#include \"tag.h\"\n+\n /*\n * We always have a files backend and it is the default.\n */\n@@ -951,6 +954,34 @@ int refname_is_safe(const char *refname)\n \treturn 1;\n }\n \n+/*\n+ * Peel the named object; i.e., if the object is a tag, resolve the\n+ * tag recursively until a non-tag is found. If successful, store the\n+ * result to sha1 and return PEEL_PEELED. If the object is not a tag\n+ * or is not valid, return PEEL_NON_TAG or PEEL_INVALID, respectively,\n+ * and leave sha1 unchanged.\n+ */\n+enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)\n+{\n+\tstruct object *o = lookup_unknown_object(name);\n+\n+\tif (o->type == OBJ_NONE) {\n+\t\tint type = sha1_object_info(name, NULL);\n+\t\tif (type < 0 || !object_as_type(o, type, 0))\n+\t\t\treturn PEEL_INVALID;\n+\t}\n+\n+\tif (o->type != OBJ_TAG)\n+\t\treturn PEEL_NON_TAG;\n+\n+\to = deref_tag_noverify(o);\n+\tif (!o)\n+\t\treturn PEEL_INVALID;\n+\n+\thashcpy(sha1, o->sha1);\n+\treturn PEEL_PEELED;\n+}\n+\n /* backend functions */\n struct ref_transaction *ref_transaction_begin(struct strbuf *err)\n {\ndiff --git a/refs.h b/refs.h\nindex c224f0e..60a98cf 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -76,6 +76,33 @@ extern int is_branch(const char *refname);\n */\n extern int peel_ref(const char *refname, unsigned char *sha1);\n \n+enum peel_status {\n+\t/* object was peeled successfully: */\n+\tPEEL_PEELED = 0,\n+\n+\t/*\n+\t * object cannot be peeled because the named object (or an\n+\t * object referred to by a tag in the peel chain), does not\n+\t * exist.\n+\t */\n+\tPEEL_INVALID = -1,\n+\n+\t/* object cannot be peeled because it is not a tag: */\n+\tPEEL_NON_TAG = -2,\n+\n+\t/* ref_entry contains no peeled value because it is a symref: */\n+\tPEEL_IS_SYMREF = -3,\n+\n+\t/*\n+\t * ref_entry cannot be peeled because it is broken (i.e., the\n+\t * symbolic reference cannot even be resolved to an object\n+\t * name):\n+\t */\n+\tPEEL_BROKEN = -4\n+};\n+\n+enum peel_status peel_object(const unsigned char *name, unsigned char *sha1);\n+\n /**\n * Resolve refname in the nested \"gitlink\" repository that is located\n * at path. If the resolution is successful, return 0 and set sha1 to\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270854","messageId":"1443477738-32023-28-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 27/43] refs.c: move should_autocreate_reflog to common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:02Z","receivedAt":"2015-09-28T22:02:02Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: David Turner \n---\n refs-be-files.c | 10 ----------\n refs.c | 10 ++++++++++\n refs.h | 2 ++\n 3 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 5971526..37e244a 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2631,16 +2631,6 @@ static int commit_ref(struct ref_lock *lock)\n \treturn 0;\n }\n \n-static int should_autocreate_reflog(const char *refname)\n-{\n-\tif (!log_all_ref_updates)\n-\t\treturn 0;\n-\treturn starts_with(refname, \"refs/heads/\") ||\n-\t\tstarts_with(refname, \"refs/remotes/\") ||\n-\t\tstarts_with(refname, \"refs/notes/\") ||\n-\t\t!strcmp(refname, \"HEAD\");\n-}\n-\n static int files_verify_refname_available(const char *newname,\n \t\t\t\t\t struct string_list *extra,\n \t\t\t\t\t struct string_list *skip,\ndiff --git a/refs.c b/refs.c\nindex d13b854..769574d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -685,6 +685,16 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t\t\t sha1, flags));\n }\n \n+int should_autocreate_reflog(const char *refname)\n+{\n+\tif (!log_all_ref_updates)\n+\t\treturn 0;\n+\treturn starts_with(refname, \"refs/heads/\") ||\n+\t\tstarts_with(refname, \"refs/remotes/\") ||\n+\t\tstarts_with(refname, \"refs/notes/\") ||\n+\t\t!strcmp(refname, \"HEAD\");\n+}\n+\n /*\n * How to handle various characters in refnames:\n * 0: An acceptable character for refs\ndiff --git a/refs.h b/refs.h\nindex 60a98cf..0b407b2 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -58,6 +58,8 @@ extern const char *resolve_ref_unsafe(const char *refname, int resolve_flags,\n extern char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags);\n \n+extern int should_autocreate_reflog(const char *refname);\n+\n extern int read_ref_full(const char *refname, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags);\n extern int read_ref(const char *refname, unsigned char *sha1);\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270859","messageId":"1443477738-32023-29-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 28/43] refs.c: add ref backend init function","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:03Z","receivedAt":"2015-09-28T22:02:03Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The file backend doesn't need this function, but other backends might.\n\nSigned-off-by: David Turner \nSigned-off-by: Ronnie Sahlberg \n---\n refs-be-files.c | 1 +\n refs.c | 4 +++-\n refs.h | 4 +++-\n 3 files changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 37e244a..eaa74b6 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3737,6 +3737,7 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n struct ref_be refs_be_files = {\n \tNULL,\n \t\"files\",\n+\tNULL,\n \tfiles_transaction_begin,\n \tfiles_transaction_update,\n \tfiles_transaction_create,\ndiff --git a/refs.c b/refs.c\nindex 769574d..9ce10b7 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,13 +19,15 @@ struct ref_be *refs_backends = &refs_be_files;\n /*\n * This function is used to switch to an alternate backend.\n */\n-int set_refs_backend(const char *name)\n+int set_refs_backend(const char *name, void *init_data)\n {\n \tstruct ref_be *be;\n \n \tfor (be = refs_backends; be; be = be->next)\n \t\tif (!strcmp(be->name, name)) {\n \t\t\tthe_refs_backend = be;\n+\t\t\tif (be->init_backend)\n+\t\t\t\tbe->init_backend(init_data);\n \t\t\treturn 0;\n \t\t}\n \treturn 1;\ndiff --git a/refs.h b/refs.h\nindex 0b407b2..0dc626e 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -586,6 +586,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t void *policy_cb_data);\n \n /* refs backends */\n+typedef void (*ref_backend_init_fn)(void *data);\n typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n typedef int (*ref_transaction_update_fn)(struct ref_transaction *transaction,\n \t\tconst char *refname, const unsigned char *new_sha1,\n@@ -641,6 +642,7 @@ typedef int (*for_each_reftype_fullpath_fn)(each_ref_fn fn, char *type,\n struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n+\tref_backend_init_fn init_backend;\n \tref_transaction_begin_fn transaction_begin;\n \tref_transaction_update_fn transaction_update;\n \tref_transaction_create_fn transaction_create;\n@@ -669,6 +671,6 @@ struct ref_be {\n \n \n extern struct ref_be refs_be_files;\n-int set_refs_backend(const char *name);\n+int set_refs_backend(const char *name, void *init_data);\n \n #endif /* REFS_H */\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270865","messageId":"1443477738-32023-30-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 29/43] refs.c: add methods for reflog","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:04Z","receivedAt":"2015-09-28T22:02:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In the file-based backend, the reflog piggybacks on the ref lock.\nSince other backends won't have the same sort of ref lock, ref backends\nmust also handle reflogs.\n\nSigned-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 32 +++++++++++++++++++++-----------\n refs.c | 34 ++++++++++++++++++++++++++++++++++\n refs.h | 23 +++++++++++++++++++++++\n 3 files changed, 78 insertions(+), 11 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex eaa74b6..fdb47e0 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2691,7 +2691,8 @@ static int log_ref_setup(const char *refname, struct strbuf *logfile, struct str\n }\n \n \n-int safe_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+static int files_create_reflog(const char *refname, int force_create,\n+\t\t\t struct strbuf *err)\n {\n \tint ret;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -2940,7 +2941,7 @@ static int files_create_symref(struct ref_transaction *trans,\n \treturn 0;\n }\n \n-int reflog_exists(const char *refname)\n+static int files_reflog_exists(const char *refname)\n {\n \tstruct stat st;\n \n@@ -2948,7 +2949,7 @@ int reflog_exists(const char *refname)\n \t\tS_ISREG(st.st_mode);\n }\n \n-int delete_reflog(const char *refname)\n+static int files_delete_reflog(const char *refname)\n {\n \treturn remove_path(git_path(\"logs/%s\", refname));\n }\n@@ -2992,7 +2993,9 @@ static char *find_beginning_of_line(char *bob, char *scan)\n \treturn scan;\n }\n \n-int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tFILE *logfp;\n@@ -3094,7 +3097,8 @@ int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void\n \treturn ret;\n }\n \n-int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data)\n+static int files_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn, void *cb_data)\n {\n \tFILE *logfp;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -3156,7 +3160,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data\n \treturn retval;\n }\n \n-int for_each_reflog(each_ref_fn fn, void *cb_data)\n+static int files_for_each_reflog(each_ref_fn fn, void *cb_data)\n {\n \tint retval;\n \tstruct strbuf name;\n@@ -3630,11 +3634,11 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n }\n \n int reflog_expire(const char *refname, const unsigned char *sha1,\n-\t\t unsigned int flags,\n-\t\t reflog_expiry_prepare_fn prepare_fn,\n-\t\t reflog_expiry_should_prune_fn should_prune_fn,\n-\t\t reflog_expiry_cleanup_fn cleanup_fn,\n-\t\t void *policy_cb_data)\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n {\n \tstatic struct lock_file reflog_lock;\n \tstruct expire_reflog_cb cb;\n@@ -3745,6 +3749,12 @@ struct ref_be refs_be_files = {\n \tfiles_transaction_verify,\n \tfiles_transaction_commit,\n \tfiles_transaction_free,\n+\tfiles_for_each_reflog_ent,\n+\tfiles_for_each_reflog_ent_reverse,\n+\tfiles_for_each_reflog,\n+\tfiles_reflog_exists,\n+\tfiles_create_reflog,\n+\tfiles_delete_reflog,\n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\n \tfiles_pack_refs,\ndiff --git a/refs.c b/refs.c\nindex 9ce10b7..7b2a47b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1155,3 +1155,37 @@ int for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken,\n \treturn the_refs_backend->for_each_reftype_fullpath(fn, type, broken,\n \t\t\t\t\t\t\t cb_data);\n }\n+\n+int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,\n+\t\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent_reverse(refname, fn,\n+\t\t\t\t\t\t\t cb_data);\n+}\n+\n+int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,\n+\t\t\tvoid *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog_ent(refname, fn, cb_data);\n+}\n+\n+int for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\treturn the_refs_backend->for_each_reflog(fn, cb_data);\n+}\n+\n+int reflog_exists(const char *refname)\n+{\n+\treturn the_refs_backend->reflog_exists(refname);\n+}\n+\n+int safe_create_reflog(const char *refname, int force_create,\n+\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->create_reflog(refname, force_create, err);\n+}\n+\n+int delete_reflog(const char *refname)\n+{\n+\treturn the_refs_backend->delete_reflog(refname);\n+}\ndiff --git a/refs.h b/refs.h\nindex 0dc626e..494f8c8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -605,6 +605,21 @@ typedef int (*ref_transaction_verify_fn)(struct ref_transaction *transaction,\n typedef int (*ref_transaction_commit_fn)(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n+\n+/* reflog functions */\n+typedef int (*for_each_reflog_ent_fn)(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data);\n+typedef int (*for_each_reflog_ent_reverse_fn)(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data);\n+typedef int (*for_each_reflog_fn)(each_ref_fn fn, void *cb_data);\n+typedef int (*reflog_exists_fn)(const char *refname);\n+typedef int (*create_reflog_fn)(const char *refname, int force_create,\n+\t\t\t\tstruct strbuf *err);\n+typedef int (*delete_reflog_fn)(const char *refname);\n+\n+/* resolution functions */\n typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n \t\t\t\t\t int resolve_flags,\n \t\t\t\t\t unsigned char *sha1, int *flags);\n@@ -620,6 +635,8 @@ typedef int (*create_symref_fn)(struct ref_transaction *transaction,\n \t\t\t\tconst char *logmsg);\n typedef int (*resolve_gitlink_ref_fn)(const char *path, const char *refname,\n \t\t\t\t unsigned char *sha1);\n+\n+/* iteration functions */\n typedef int (*head_ref_fn)(each_ref_fn fn, void *cb_data);\n typedef int (*head_ref_submodule_fn)(const char *submodule, each_ref_fn fn,\n \t\t\t\t void *cb_data);\n@@ -650,6 +667,12 @@ struct ref_be {\n \tref_transaction_verify_fn transaction_verify;\n \tref_transaction_commit_fn transaction_commit;\n \tref_transaction_free_fn transaction_free;\n+\tfor_each_reflog_ent_fn for_each_reflog_ent;\n+\tfor_each_reflog_ent_reverse_fn for_each_reflog_ent_reverse;\n+\tfor_each_reflog_fn for_each_reflog;\n+\treflog_exists_fn reflog_exists;\n+\tcreate_reflog_fn create_reflog;\n+\tdelete_reflog_fn delete_reflog;\n \tresolve_ref_unsafe_fn resolve_ref_unsafe;\n \tverify_refname_available_fn verify_refname_available;\n \tpack_refs_fn pack_refs;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270862","messageId":"1443477738-32023-31-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 30/43] refs-be-files.c: add method to expire reflogs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:05Z","receivedAt":"2015-09-28T22:02:05Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: David Turner \n---\n refs-be-files.c | 13 +++++++------\n refs.c | 12 ++++++++++++\n refs.h | 7 +++++++\n 3 files changed, 26 insertions(+), 6 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex fdb47e0..93a5a33 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3633,12 +3633,12 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \treturn 0;\n }\n \n-int reflog_expire(const char *refname, const unsigned char *sha1,\n-\t\t unsigned int flags,\n-\t\t reflog_expiry_prepare_fn prepare_fn,\n-\t\t reflog_expiry_should_prune_fn should_prune_fn,\n-\t\t reflog_expiry_cleanup_fn cleanup_fn,\n-\t\t void *policy_cb_data)\n+static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t\t unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n {\n \tstatic struct lock_file reflog_lock;\n \tstruct expire_reflog_cb cb;\n@@ -3755,6 +3755,7 @@ struct ref_be refs_be_files = {\n \tfiles_reflog_exists,\n \tfiles_create_reflog,\n \tfiles_delete_reflog,\n+\tfiles_reflog_expire,\n \tfiles_resolve_ref_unsafe,\n \tfiles_verify_refname_available,\n \tfiles_pack_refs,\ndiff --git a/refs.c b/refs.c\nindex 7b2a47b..96fa699 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1189,3 +1189,15 @@ int delete_reflog(const char *refname)\n {\n \treturn the_refs_backend->delete_reflog(refname);\n }\n+\n+int reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\treturn the_refs_backend->reflog_expire(refname, sha1, flags,\n+\t\t\t\t\t prepare_fn, should_prune_fn,\n+\t\t\t\t\t cleanup_fn, policy_cb_data);\n+}\ndiff --git a/refs.h b/refs.h\nindex 494f8c8..06089f8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -618,6 +618,12 @@ typedef int (*reflog_exists_fn)(const char *refname);\n typedef int (*create_reflog_fn)(const char *refname, int force_create,\n \t\t\t\tstruct strbuf *err);\n typedef int (*delete_reflog_fn)(const char *refname);\n+typedef int (*reflog_expire_fn)(const char *refname, const unsigned char *sha1,\n+\t\t\t\tunsigned int flags,\n+\t\t\t\treflog_expiry_prepare_fn prepare_fn,\n+\t\t\t\treflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t\treflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t\tvoid *policy_cb_data);\n \n /* resolution functions */\n typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n@@ -673,6 +679,7 @@ struct ref_be {\n \treflog_exists_fn reflog_exists;\n \tcreate_reflog_fn create_reflog;\n \tdelete_reflog_fn delete_reflog;\n+\treflog_expire_fn reflog_expire;\n \tresolve_ref_unsafe_fn resolve_ref_unsafe;\n \tverify_refname_available_fn verify_refname_available;\n \tpack_refs_fn pack_refs;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270867","messageId":"1443477738-32023-32-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 31/43] refs.c: add method for initial ref transaction commit","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:06Z","receivedAt":"2015-09-28T22:02:06Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: Ronnie Sahlberg \nSigned-off-by: David Turner \n---\n refs-be-files.c | 5 +++--\n refs.c | 6 ++++++\n refs.h | 1 +\n 3 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 93a5a33..ea91afd 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3517,8 +3517,8 @@ static int ref_present(const char *refname,\n \treturn string_list_has_string(affected_refnames, refname);\n }\n \n-int initial_ref_transaction_commit(struct ref_transaction *transaction,\n-\t\t\t\t struct strbuf *err)\n+static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n \tint n = transaction->nr;\n@@ -3748,6 +3748,7 @@ struct ref_be refs_be_files = {\n \tfiles_transaction_delete,\n \tfiles_transaction_verify,\n \tfiles_transaction_commit,\n+\tfiles_initial_transaction_commit,\n \tfiles_transaction_free,\n \tfiles_for_each_reflog_ent,\n \tfiles_for_each_reflog_ent_reverse,\ndiff --git a/refs.c b/refs.c\nindex 96fa699..530edcf 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1201,3 +1201,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t\t\t prepare_fn, should_prune_fn,\n \t\t\t\t\t cleanup_fn, policy_cb_data);\n }\n+\n+int initial_ref_transaction_commit(struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn the_refs_backend->initial_transaction_commit(transaction, err);\n+}\ndiff --git a/refs.h b/refs.h\nindex 06089f8..bd46d0f 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -672,6 +672,7 @@ struct ref_be {\n \tref_transaction_delete_fn transaction_delete;\n \tref_transaction_verify_fn transaction_verify;\n \tref_transaction_commit_fn transaction_commit;\n+\tref_transaction_commit_fn initial_transaction_commit;\n \tref_transaction_free_fn transaction_free;\n \tfor_each_reflog_ent_fn for_each_reflog_ent;\n \tfor_each_reflog_ent_reverse_fn for_each_reflog_ent_reverse;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270853","messageId":"1443477738-32023-33-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 32/43] initdb: move safe_create_dir into common code","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:07Z","receivedAt":"2015-09-28T22:02:07Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"In a moment, we'll create initdb functions for ref backends, and code\nfrom initdb that calls this function needs to move into the files\nbackend. So this function needs to be public.\n\nSigned-off-by: David Turner \n---\n builtin/init-db.c | 12 ------------\n cache.h | 5 +++++\n path.c | 12 ++++++++++++\n 3 files changed, 17 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex cf6a3c8..f0b095c 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -24,18 +24,6 @@ static int init_shared_repository = -1;\n static const char *init_db_template_dir;\n static const char *git_link;\n \n-static void safe_create_dir(const char *dir, int share)\n-{\n-\tif (mkdir(dir, 0777) < 0) {\n-\t\tif (errno != EEXIST) {\n-\t\t\tperror(dir);\n-\t\t\texit(1);\n-\t\t}\n-\t}\n-\telse if (share && adjust_shared_perm(dir))\n-\t\tdie(_(\"Could not make %s writable by group\"), dir);\n-}\n-\n static void copy_templates_1(struct strbuf *path, struct strbuf *template,\n \t\t\t DIR *dir)\n {\ndiff --git a/cache.h b/cache.h\nindex 1c45b62..cc817dc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1749,4 +1749,9 @@ void stat_validity_update(struct stat_validity *sv, int fd);\n int versioncmp(const char *s1, const char *s2);\n void sleep_millisec(int millisec);\n \n+/*\n+ * Create a directory and (if share is nonzero) adjust its permissions\n+ * according to the shared_repository setting.\n+ */\n+void safe_create_dir(const char *dir, int share);\n #endif /* CACHE_H */\ndiff --git a/path.c b/path.c\nindex 48bd252..9ab2710 100644\n--- a/path.c\n+++ b/path.c\n@@ -740,6 +740,18 @@ int adjust_shared_perm(const char *path)\n \treturn 0;\n }\n \n+void safe_create_dir(const char *dir, int share)\n+{\n+\tif (mkdir(dir, 0777) < 0) {\n+\t\tif (errno != EEXIST) {\n+\t\t\tperror(dir);\n+\t\t\texit(1);\n+\t\t}\n+\t}\n+\telse if (share && adjust_shared_perm(dir))\n+\t\tdie(_(\"Could not make %s writable by group\"), dir);\n+}\n+\n static int have_same_root(const char *path1, const char *path2)\n {\n \tint is_abs1, is_abs2;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270869","messageId":"1443477738-32023-34-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 33/43] refs.c: add method for initializing refs db","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:08Z","receivedAt":"2015-09-28T22:02:08Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate refs backends might not need refs/heads and so on, so we make\nref db initialization part of the backend.\n\nSigned-off-by: David Turner \n---\n builtin/init-db.c | 14 ++++----------\n refs-be-files.c | 17 +++++++++++++++++\n refs.c | 5 +++++\n refs.h | 4 ++++\n 4 files changed, 30 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex f0b095c..504a2dc 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -177,13 +177,7 @@ static int create_default_files(const char *template_path)\n \tchar junk[2];\n \tint reinit;\n \tint filemode;\n-\n-\t/*\n-\t * Create .git/refs/{heads,tags}\n-\t */\n-\tsafe_create_dir(git_path_buf(&buf, \"refs\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/heads\"), 1);\n-\tsafe_create_dir(git_path_buf(&buf, \"refs/tags\"), 1);\n+\tstruct strbuf err = STRBUF_INIT;\n \n \t/* Just look for `init.templatedir` */\n \tgit_config(git_init_db_config, NULL);\n@@ -207,11 +201,11 @@ static int create_default_files(const char *template_path)\n \t */\n \tif (shared_repository) {\n \t\tadjust_shared_perm(get_git_dir());\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/heads\"));\n-\t\tadjust_shared_perm(git_path_buf(&buf, \"refs/tags\"));\n \t}\n \n+\tif (refs_initdb(&err, shared_repository))\n+\t\tdie(\"failed to set up refs db: %s\", err.buf);\n+\n \t/*\n \t * Create the default symlink from \".git/HEAD\" to the \"master\"\n \t * branch, if it does not exist yet.\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex ea91afd..0a76c8e 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3738,10 +3738,27 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n \treturn -1;\n }\n \n+static int files_initdb(struct strbuf *err, int shared)\n+{\n+\t/*\n+\t * Create .git/refs/{heads,tags}\n+\t */\n+\tsafe_create_dir(git_path(\"refs\"), 1);\n+\tsafe_create_dir(git_path(\"refs/heads\"), 1);\n+\tsafe_create_dir(git_path(\"refs/tags\"), 1);\n+\tif (shared) {\n+\t\tadjust_shared_perm(git_path(\"refs\"));\n+\t\tadjust_shared_perm(git_path(\"refs/heads\"));\n+\t\tadjust_shared_perm(git_path(\"refs/tags\"));\n+\t}\n+\treturn 0;\n+}\n+\n struct ref_be refs_be_files = {\n \tNULL,\n \t\"files\",\n \tNULL,\n+\tfiles_initdb,\n \tfiles_transaction_begin,\n \tfiles_transaction_update,\n \tfiles_transaction_create,\ndiff --git a/refs.c b/refs.c\nindex 530edcf..930c85c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -995,6 +995,11 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)\n }\n \n /* backend functions */\n+int refs_initdb(struct strbuf *err, int shared)\n+{\n+\treturn the_refs_backend->initdb(err, shared);\n+}\n+\n struct ref_transaction *ref_transaction_begin(struct strbuf *err)\n {\n \treturn the_refs_backend->transaction_begin(err);\ndiff --git a/refs.h b/refs.h\nindex bd46d0f..64dba64 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -68,6 +68,8 @@ extern int ref_exists(const char *refname);\n \n extern int is_branch(const char *refname);\n \n+extern int refs_initdb(struct strbuf *err, int shared);\n+\n /*\n * If refname is a non-symbolic reference that refers to a tag object,\n * and the tag can be (recursively) dereferenced to a non-tag object,\n@@ -587,6 +589,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \n /* refs backends */\n typedef void (*ref_backend_init_fn)(void *data);\n+typedef int (*ref_backend_initdb_fn)(struct strbuf *err, int shared);\n typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n typedef int (*ref_transaction_update_fn)(struct ref_transaction *transaction,\n \t\tconst char *refname, const unsigned char *new_sha1,\n@@ -666,6 +669,7 @@ struct ref_be {\n \tstruct ref_be *next;\n \tconst char *name;\n \tref_backend_init_fn init_backend;\n+\tref_backend_initdb_fn initdb;\n \tref_transaction_begin_fn transaction_begin;\n \tref_transaction_update_fn transaction_update;\n \tref_transaction_create_fn transaction_create;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270868","messageId":"1443477738-32023-35-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 34/43] refs.c: make struct ref_transaction generic","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:09Z","receivedAt":"2015-09-28T22:02:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Alternate ref backends might need different data for transactions. Make\nstruct ref_transaction an empty struct, and let backends define their\nown structs which extend it.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 24 +++++++++++++++++-------\n refs.h | 8 ++++++--\n 2 files changed, 23 insertions(+), 9 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 0a76c8e..3f2d194 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -3218,7 +3218,8 @@ enum ref_transaction_state {\n * consist of checks and updates to multiple references, carried out\n * as atomically as possible. This structure is opaque to callers.\n */\n-struct ref_transaction {\n+struct files_ref_transaction {\n+\tstruct ref_transaction base;\n \tstruct ref_update **updates;\n \tsize_t alloc;\n \tsize_t nr;\n@@ -3229,13 +3230,16 @@ static struct ref_transaction *files_transaction_begin(struct strbuf *err)\n {\n \tassert(err);\n \n-\treturn xcalloc(1, sizeof(struct ref_transaction));\n+\treturn xcalloc(1, sizeof(struct files_ref_transaction));\n }\n \n-static void files_transaction_free(struct ref_transaction *transaction)\n+static void files_transaction_free(struct ref_transaction *trans)\n {\n \tint i;\n \n+\tstruct files_ref_transaction *transaction =\n+\t\t(struct files_ref_transaction *)trans;\n+\n \tif (!transaction)\n \t\treturn;\n \n@@ -3247,7 +3251,7 @@ static void files_transaction_free(struct ref_transaction *transaction)\n \tfree(transaction);\n }\n \n-static struct ref_update *add_update(struct ref_transaction *transaction,\n+static struct ref_update *add_update(struct files_ref_transaction *transaction,\n \t\t\t\t const char *refname)\n {\n \tsize_t len = strlen(refname) + 1;\n@@ -3259,7 +3263,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n \treturn update;\n }\n \n-static int files_transaction_update(struct ref_transaction *transaction,\n+static int files_transaction_update(struct ref_transaction *trans,\n \t\t\t\t const char *refname,\n \t\t\t\t const unsigned char *new_sha1,\n \t\t\t\t const unsigned char *old_sha1,\n@@ -3267,6 +3271,8 @@ static int files_transaction_update(struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n {\n \tstruct ref_update *update;\n+\tstruct files_ref_transaction *transaction =\n+\t\t(struct files_ref_transaction *)trans;\n \n \tassert(err);\n \n@@ -3350,10 +3356,12 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int files_transaction_commit(struct ref_transaction *transaction,\n+static int files_transaction_commit(struct ref_transaction *trans,\n \t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n+\tstruct files_ref_transaction *transaction =\n+\t\t(struct files_ref_transaction *)trans;\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n \tstruct string_list refs_to_delete = STRING_LIST_INIT_NODUP;\n@@ -3517,10 +3525,12 @@ static int ref_present(const char *refname,\n \treturn string_list_has_string(affected_refnames, refname);\n }\n \n-static int files_initial_transaction_commit(struct ref_transaction *transaction,\n+static int files_initial_transaction_commit(struct ref_transaction *trans,\n \t\t\t\t\t struct strbuf *err)\n {\n \tint ret = 0, i;\n+\tstruct files_ref_transaction *transaction =\n+\t\t(struct files_ref_transaction *)trans;\n \tint n = transaction->nr;\n \tstruct ref_update **updates = transaction->updates;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\ndiff --git a/refs.h b/refs.h\nindex 64dba64..02001ef 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -130,7 +130,7 @@ extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);\n *\n * Calling sequence\n * ----------------\n- * - Allocate and initialize a `struct ref_transaction` by calling\n+ * - Allocate and initialize a transaction by calling\n * `ref_transaction_begin()`.\n *\n * - List intended ref updates by calling functions like\n@@ -156,7 +156,10 @@ extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);\n * The message is appended to err without first clearing err.\n * err will not be '\\n' terminated.\n */\n-struct ref_transaction;\n+\n+struct ref_transaction {\n+\t/* ref backends should extend this */\n+};\n \n /*\n * Bit values set in the flags argument passed to each_ref_fn():\n@@ -629,6 +632,7 @@ typedef int (*reflog_expire_fn)(const char *refname, const unsigned char *sha1,\n \t\t\t\tvoid *policy_cb_data);\n \n /* resolution functions */\n+typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n \t\t\t\t\t int resolve_flags,\n \t\t\t\t\t unsigned char *sha1, int *flags);\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270856","messageId":"1443477738-32023-36-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 35/43] refs-be-files.c: add method to rename refs","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:10Z","receivedAt":"2015-09-28T22:02:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"We also have to make rename_ref_available public, since alternate\nbackends for rename_ref will need it.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 20 +++-----------------\n refs.c | 21 +++++++++++++++++++++\n refs.h | 7 ++++++-\n 3 files changed, 30 insertions(+), 18 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 3f2d194..3bcfab3 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2491,29 +2491,14 @@ out:\n \treturn ret;\n }\n \n-static int rename_ref_available(const char *oldname, const char *newname)\n-{\n-\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n-\tstruct strbuf err = STRBUF_INIT;\n-\tint ret;\n-\n-\tstring_list_insert(&skip, oldname);\n-\tret = !verify_refname_available(newname, NULL, &skip, &err);\n-\tif (!ret)\n-\t\terror(\"%s\", err.buf);\n-\n-\tstring_list_clear(&skip, 0);\n-\tstrbuf_release(&err);\n-\treturn ret;\n-}\n-\n static int write_ref_to_lockfile(struct ref_lock *lock,\n \t\t\t\t const unsigned char *sha1, struct strbuf *err);\n static int commit_ref_update(struct ref_lock *lock,\n \t\t\t const unsigned char *sha1, const char *logmsg,\n \t\t\t int flags, struct strbuf *err);\n \n-int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)\n+static int files_rename_ref(const char *oldrefname, const char *newrefname,\n+\t\t\t const char *logmsg)\n {\n \tunsigned char sha1[20], orig_sha1[20];\n \tint flag = 0, logmoved = 0;\n@@ -3777,6 +3762,7 @@ struct ref_be refs_be_files = {\n \tfiles_transaction_commit,\n \tfiles_initial_transaction_commit,\n \tfiles_transaction_free,\n+\tfiles_rename_ref,\n \tfiles_for_each_reflog_ent,\n \tfiles_for_each_reflog_ent_reverse,\n \tfiles_for_each_reflog,\ndiff --git a/refs.c b/refs.c\nindex 930c85c..1c2dd79 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1052,6 +1052,11 @@ void ref_transaction_free(struct ref_transaction *transaction)\n \treturn the_refs_backend->transaction_free(transaction);\n }\n \n+int rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\treturn the_refs_backend->rename_ref(oldref, newref, logmsg);\n+}\n+\n const char *resolve_ref_unsafe(const char *ref, int resolve_flags,\n \t\t\t unsigned char *sha1, int *flags)\n {\n@@ -1066,6 +1071,22 @@ int verify_refname_available(const char *refname, struct string_list *extra,\n \t\t\t\t\t\t\t err);\n }\n \n+int rename_ref_available(const char *oldname, const char *newname)\n+{\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint ret;\n+\n+\tstring_list_insert(&skip, oldname);\n+\tret = !verify_refname_available(newname, NULL, &skip, &err);\n+\tif (!ret)\n+\t\terror(\"%s\", err.buf);\n+\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&err);\n+\treturn ret;\n+}\n+\n int pack_refs(unsigned int flags)\n {\n \treturn the_refs_backend->pack_refs(flags);\ndiff --git a/refs.h b/refs.h\nindex 02001ef..823983b 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -378,10 +378,13 @@ extern const char *prettify_refname(const char *refname);\n extern char *shorten_unambiguous_ref(const char *refname, int strict);\n \n /** rename ref, return 0 on success **/\n-extern int rename_ref(const char *oldref, const char *newref, const char *logmsg);\n+extern int rename_ref(const char *oldref, const char *newref,\n+\t\t const char *logmsg);\n \n extern int create_symref(const char *ref, const char *refs_heads_master, const char *logmsg);\n \n+int rename_ref_available(const char *oldname, const char *newname);\n+\n enum action_on_err {\n \tUPDATE_REFS_MSG_ON_ERR,\n \tUPDATE_REFS_DIE_ON_ERR,\n@@ -633,6 +636,7 @@ typedef int (*reflog_expire_fn)(const char *refname, const unsigned char *sha1,\n \n /* resolution functions */\n typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n+typedef int (*rename_ref_fn)(const char *oldref, const char *newref, const char *logmsg);\n typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n \t\t\t\t\t int resolve_flags,\n \t\t\t\t\t unsigned char *sha1, int *flags);\n@@ -682,6 +686,7 @@ struct ref_be {\n \tref_transaction_commit_fn transaction_commit;\n \tref_transaction_commit_fn initial_transaction_commit;\n \tref_transaction_free_fn transaction_free;\n+\trename_ref_fn rename_ref;\n \tfor_each_reflog_ent_fn for_each_reflog_ent;\n \tfor_each_reflog_ent_reverse_fn for_each_reflog_ent_reverse;\n \tfor_each_reflog_fn for_each_reflog;\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270866","messageId":"1443477738-32023-37-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 36/43] run-command: track total number of commands run","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:11Z","receivedAt":"2015-09-28T22:02:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Soon, the db refs backend will use this to restart read transactions\nafter subcommands which might have modified the refs database.\n\nSigned-off-by: David Turner \n---\n run-command.c | 2 ++\n run-command.h | 1 +\n 2 files changed, 3 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex ceee5d1..9ee6c85 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -19,6 +19,7 @@ struct child_to_clean {\n };\n static struct child_to_clean *children_to_clean;\n static int installed_child_cleanup_handler;\n+int total_commands_run = 0;\n \n static void cleanup_children(int sig, int in_signal)\n {\n@@ -349,6 +350,7 @@ fail_pipe:\n \t\tcmd->err = fderr[0];\n \t}\n \n+\ttotal_commands_run++;\n \ttrace_argv_printf(cmd->argv, \"trace: run_command:\");\n \tfflush(NULL);\n \ndiff --git a/run-command.h b/run-command.h\nindex aa2a528..ef7ae7a 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -158,4 +158,5 @@ int run_processes_parallel(int n, void *data,\n \t\t\t start_failure_fn,\n \t\t\t return_value_fn);\n \n+extern int total_commands_run;\n #endif\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270864","messageId":"1443477738-32023-38-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 37/43] refs: move some defines from refs-be-files.c to refs.h","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:12Z","receivedAt":"2015-09-28T22:02:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This allows them to be used by other ref backends.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 24 ------------------------\n refs.h | 24 ++++++++++++++++++++++++\n 2 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 3bcfab3..2727943 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -14,30 +14,6 @@ struct ref_lock {\n };\n \n /*\n- * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken\n- * refs (i.e., because the reference is about to be deleted anyway).\n- */\n-#define REF_DELETING\t0x02\n-\n-/*\n- * Used as a flag in ref_update::flags when a loose ref is being\n- * pruned.\n- */\n-#define REF_ISPRUNING\t0x04\n-\n-/*\n- * Used as a flag in ref_update::flags when the reference should be\n- * updated to new_sha1.\n- */\n-#define REF_HAVE_NEW\t0x08\n-\n-/*\n- * Used as a flag in ref_update::flags when old_sha1 should be\n- * checked.\n- */\n-#define REF_HAVE_OLD\t0x10\n-\n-/*\n * Used as a flag in ref_update::flags when the lockfile needs to be\n * committed.\n */\ndiff --git a/refs.h b/refs.h\nindex 823983b..d8ae9af 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -193,6 +193,30 @@ struct ref_transaction {\n #define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n \n /*\n+ * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken\n+ * refs (i.e., because the reference is about to be deleted anyway).\n+ */\n+#define REF_DELETING\t0x02\n+\n+/*\n+ * Used as a flag in ref_update::flags when a loose ref is being\n+ * pruned.\n+ */\n+#define REF_ISPRUNING\t0x04\n+\n+/*\n+ * Used as a flag in ref_update::flags when the reference should be\n+ * updated to new_sha1.\n+ */\n+#define REF_HAVE_NEW\t0x08\n+\n+/*\n+ * Used as a flag in ref_update::flags when old_sha1 should be\n+ * checked.\n+ */\n+#define REF_HAVE_OLD\t0x10\n+\n+/*\n * The signature for the callback function for the for_each_*()\n * functions below. The memory pointed to by the refname and sha1\n * arguments is only guaranteed to be valid for the duration of a\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270863","messageId":"1443477738-32023-39-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 38/43] refs: make some files backend functions public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:13Z","receivedAt":"2015-09-28T22:02:13Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Because HEAD and stash are per-worktree, other backends need to\ngo through the files backend to manage these refs and their reflogs.\n\nTo enable this, we make some files backend functions public.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 34 +++++++++++++++++++++-------------\n refs.h | 20 ++++++++++++++++++++\n 2 files changed, 41 insertions(+), 13 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 2727943..943604c 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -2727,6 +2727,14 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,\n \t\t\t const unsigned char *new_sha1, const char *msg,\n \t\t\t int flags, struct strbuf *err)\n {\n+\treturn files_log_ref_write(refname, old_sha1, new_sha1, msg, flags,\n+\t\t\t\t err);\n+}\n+\n+int files_log_ref_write(const char *refname, const unsigned char *old_sha1,\n+\t\t\tconst unsigned char *new_sha1, const char *msg,\n+\t\t\tint flags, struct strbuf *err)\n+{\n \tstruct strbuf sb = STRBUF_INIT;\n \tint ret = log_ref_write_1(refname, old_sha1, new_sha1, msg, &sb, flags,\n \t\t\t\t err);\n@@ -2902,7 +2910,7 @@ static int files_create_symref(struct ref_transaction *trans,\n \treturn 0;\n }\n \n-static int files_reflog_exists(const char *refname)\n+int files_reflog_exists(const char *refname)\n {\n \tstruct stat st;\n \n@@ -2910,7 +2918,7 @@ static int files_reflog_exists(const char *refname)\n \t\tS_ISREG(st.st_mode);\n }\n \n-static int files_delete_reflog(const char *refname)\n+int files_delete_reflog(const char *refname)\n {\n \treturn remove_path(git_path(\"logs/%s\", refname));\n }\n@@ -2954,9 +2962,9 @@ static char *find_beginning_of_line(char *bob, char *scan)\n \treturn scan;\n }\n \n-static int files_for_each_reflog_ent_reverse(const char *refname,\n-\t\t\t\t\t each_reflog_ent_fn fn,\n-\t\t\t\t\t void *cb_data)\n+int files_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tFILE *logfp;\n@@ -3058,8 +3066,8 @@ static int files_for_each_reflog_ent_reverse(const char *refname,\n \treturn ret;\n }\n \n-static int files_for_each_reflog_ent(const char *refname,\n-\t\t\t\t each_reflog_ent_fn fn, void *cb_data)\n+int files_for_each_reflog_ent(const char *refname,\n+\t\t\t each_reflog_ent_fn fn, void *cb_data)\n {\n \tFILE *logfp;\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -3604,12 +3612,12 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \treturn 0;\n }\n \n-static int files_reflog_expire(const char *refname, const unsigned char *sha1,\n-\t\t\t unsigned int flags,\n-\t\t\t reflog_expiry_prepare_fn prepare_fn,\n-\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n-\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n-\t\t\t void *policy_cb_data)\n+int files_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t\tunsigned int flags,\n+\t\t\treflog_expiry_prepare_fn prepare_fn,\n+\t\t\treflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\treflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\tvoid *policy_cb_data)\n {\n \tstatic struct lock_file reflog_lock;\n \tstruct expire_reflog_cb cb;\ndiff --git a/refs.h b/refs.h\nindex d8ae9af..da29232 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -571,6 +571,10 @@ enum ref_type ref_type(const char *refname);\n \n int copy_reflog_msg(char *buf, const char *msg);\n \n+int files_log_ref_write(const char *refname, const unsigned char *old_sha1,\n+\t\t\tconst unsigned char *new_sha1, const char *msg,\n+\t\t\tint flags, struct strbuf *err);\n+\n enum expire_reflog_flags {\n \tEXPIRE_REFLOGS_DRY_RUN = 1 << 0,\n \tEXPIRE_REFLOGS_UPDATE_REF = 1 << 1,\n@@ -604,6 +608,22 @@ typedef int reflog_expiry_should_prune_fn(unsigned char *osha1,\n \t\t\t\t\t const char *message, void *cb_data);\n typedef void reflog_expiry_cleanup_fn(void *cb_data);\n \n+int files_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t\tunsigned int flags,\n+\t\t\treflog_expiry_prepare_fn prepare_fn,\n+\t\t\treflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\treflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\tvoid *policy_cb_data);\n+\n+int files_for_each_reflog_ent(const char *refname,\n+\t\t\t each_reflog_ent_fn fn, void *cb_data);\n+\n+int files_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn, void *cb_data);\n+\n+int files_delete_reflog(const char *refname);\n+int files_reflog_exists(const char *refname);\n+\n /*\n * Expire reflog entries for the specified reference. sha1 is the old\n * value of the reference. flags is a combination of the constants in\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270855","messageId":"1443477738-32023-40-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 39/43] refs: break out a ref conflict check","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:14Z","receivedAt":"2015-09-28T22:02:14Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Create new function verify_no_descendants, to hold one of the ref\nconflict checks used in verify_refname_available. Multiple backends\nwill need this function, so it goes in the common code.\n\nSigned-off-by: David Turner \n---\n refs-be-files.c | 33 ++++++++-------------------------\n refs.c | 28 ++++++++++++++++++++++++++++\n refs.h | 4 ++++\n 3 files changed, 40 insertions(+), 25 deletions(-)\n\ndiff --git a/refs-be-files.c b/refs-be-files.c\nindex 943604c..8815111 100644\n--- a/refs-be-files.c\n+++ b/refs-be-files.c\n@@ -753,6 +753,7 @@ static int verify_refname_available_dir(const char *refname,\n \t\t\t\t\tstruct strbuf *err)\n {\n \tconst char *slash;\n+\tconst char *extra_refname;\n \tint pos;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tint ret = -1;\n@@ -858,33 +859,15 @@ static int verify_refname_available_dir(const char *refname,\n \t\t}\n \t}\n \n-\tif (extras) {\n-\t\t/*\n-\t\t * Check for entries in extras that start with\n-\t\t * \"$refname/\". We do that by looking for the place\n-\t\t * where \"$refname/\" would be inserted in extras. If\n-\t\t * there is an entry at that position that starts with\n-\t\t * \"$refname/\" and is not in skip, then we have a\n-\t\t * conflict.\n-\t\t */\n-\t\tfor (pos = string_list_find_insert_index(extras, dirname.buf, 0);\n-\t\t pos < extras->nr; pos++) {\n-\t\t\tconst char *extra_refname = extras->items[pos].string;\n-\n-\t\t\tif (!starts_with(extra_refname, dirname.buf))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (!skip || !string_list_has_string(skip, extra_refname)) {\n-\t\t\t\tstrbuf_addf(err, \"cannot process '%s' and '%s' at the same time\",\n-\t\t\t\t\t refname, extra_refname);\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t}\n+\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\tif (extra_refname) {\n+\t\tstrbuf_addf(err,\n+\t\t\t \"cannot process '%s' and '%s' at the same time\",\n+\t\t\t refname, extra_refname);\n+\t} else {\n+\t\tret = 0;\n \t}\n \n-\t/* No conflicts were found */\n-\tret = 0;\n-\n cleanup:\n \tstrbuf_release(&dirname);\n \treturn ret;\ndiff --git a/refs.c b/refs.c\nindex 1c2dd79..17a364a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -994,6 +994,34 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)\n \treturn PEEL_PEELED;\n }\n \n+const char *find_descendant_ref(const char *refname,\n+\t\t\t\tconst struct string_list *extras,\n+\t\t\t\tconst struct string_list *skip)\n+{\n+\tint pos;\n+\tif (!extras)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * Check for entries in extras that start with \"$refname/\". We\n+\t * do that by looking for the place where \"$refname/\" would be\n+\t * inserted in extras. If there is an entry at that position\n+\t * that starts with \"$refname/\" and is not in skip, then we\n+\t * have a conflict.\n+\t */\n+\tfor (pos = string_list_find_insert_index(extras, refname, 0);\n+\t pos < extras->nr; pos++) {\n+\t\tconst char *extra_refname = extras->items[pos].string;\n+\n+\t\tif (!starts_with(extra_refname, refname))\n+\t\t\tbreak;\n+\n+\t\tif (!skip || !string_list_has_string(skip, extra_refname))\n+\t\t\treturn extra_refname;\n+\t}\n+\treturn NULL;\n+}\n+\n /* backend functions */\n int refs_initdb(struct strbuf *err, int shared)\n {\ndiff --git a/refs.h b/refs.h\nindex da29232..cf1780e 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -571,6 +571,10 @@ enum ref_type ref_type(const char *refname);\n \n int copy_reflog_msg(char *buf, const char *msg);\n \n+const char *find_descendant_ref(const char *refname,\n+\t\t\t\tconst struct string_list *extras,\n+\t\t\t\tconst struct string_list *skip);\n+\n int files_log_ref_write(const char *refname, const unsigned char *old_sha1,\n \t\t\tconst unsigned char *new_sha1, const char *msg,\n \t\t\tint flags, struct strbuf *err);\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270858","messageId":"1443477738-32023-41-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:15Z","receivedAt":"2015-09-28T22:02:15Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a new option, --refs-backend-type, to allow the ref backend type to\nbe set on new clones.\n\nSubmodules must use the same ref backend as the parent repository, so\nwe also pass the --refs-backend-type option option when cloning\nsubmodules.\n\nSigned-off-by: David Turner \n---\n Documentation/git-clone.txt | 4 ++++\n builtin/clone.c | 27 +++++++++++++++++++++++++--\n builtin/submodule--helper.c | 5 ++++-\n cache.h | 1 +\n refs.c | 2 ++\n 5 files changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex f1f2a3f..d7a4cb0 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t [-o ] [-b ] [-u ] [--reference ]\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n+\t [--refs-backend-type=]\n \t [--recursive | --recurse-submodules] [--] \n \t []\n \n@@ -216,6 +217,9 @@ objects from the source repository into a pack in the cloned repository.\n \tThe result is Git repository can be separated from working\n \ttree.\n \n+--refs-backend-type=::\n+\tType of refs backend. Default is to use the original files based\n+\tbackend.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 3e14491..d489a87 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -92,11 +92,13 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n \t\t\tN_(\"set config inside the new repository\")),\n+\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n+\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \tOPT_END()\n };\n \n static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n+\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL, NULL\n };\n \n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n@@ -724,8 +726,24 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n+\tif (!err && option_recursive) {\n+\t\tconst char **backend_arg = argv_submodule;\n+\t\tchar *new_backend_arg = NULL;\n+\t\tif (refs_backend_type) {\n+\t\t\twhile (*backend_arg)\n+\t\t\t\t++backend_arg;\n+\n+\t\t\tnew_backend_arg = xmalloc(21 + strlen(refs_backend_type));\n+\t\t\tsprintf(new_backend_arg, \"--refs-backend-type=%s\",\n+\t\t\t\trefs_backend_type);\n+\t\t\t*backend_arg = new_backend_arg;\n+\t\t}\n \t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\t\tif (refs_backend_type) {\n+\t\t\tfree(new_backend_arg);\n+\t\t\t*backend_arg = NULL;\n+\t\t}\n+\t}\n \n \treturn err;\n }\n@@ -744,6 +762,11 @@ static void write_config(struct string_list *config)\n \t\t\t\t\t write_one_config, NULL) < 0)\n \t\t\tdie(\"unable to write parameters to config file\");\n \t}\n+\n+\tif (refs_backend_type &&\n+\t write_one_config(\"core.refs-backend-type\",\n+\t\t\t refs_backend_type, NULL) < 0)\n+\t\t\tdie(\"unable to write backend parameter to config file\");\n }\n \n static void write_refspec_config(const char *src_ref_prefix,\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex f4c3eff..5c9ca4e 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -140,7 +140,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url\n \t\targv_array_pushl(&cp.args, \"--reference\", reference, NULL);\n \tif (gitdir && *gitdir)\n \t\targv_array_pushl(&cp.args, \"--separate-git-dir\", gitdir, NULL);\n-\n+\tif (refs_backend_type && *refs_backend_type) {\n+\t\targv_array_push(&cp.args, \"--refs-backend-type\");\n+\t\targv_array_push(&cp.args, refs_backend_type);\n+\t}\n \targv_array_push(&cp.args, url);\n \targv_array_push(&cp.args, path);\n \ndiff --git a/cache.h b/cache.h\nindex cc817dc..692cfd3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -695,6 +695,7 @@ enum object_creation_mode {\n extern enum object_creation_mode object_creation_mode;\n \n extern char *notes_ref_name;\n+extern const char *refs_backend_type;\n \n extern int grafts_replace_parents;\n \ndiff --git a/refs.c b/refs.c\nindex 17a364a..4f2ab25 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -16,6 +16,8 @@ struct ref_be *the_refs_backend = &refs_be_files;\n */\n struct ref_be *refs_backends = &refs_be_files;\n \n+const char *refs_backend_type;\n+\n /*\n * This function is used to switch to an alternate backend.\n */\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270857","messageId":"1443477738-32023-42-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 41/43] refs: add register_refs_backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:16Z","receivedAt":"2015-09-28T22:02:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add register_refs_backend, to allow refs backends to be registered.\n\nSigned-off-by: David Turner \n---\n refs.c | 6 ++++++\n refs.h | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 4f2ab25..4195657 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -18,6 +18,12 @@ struct ref_be *refs_backends = &refs_be_files;\n \n const char *refs_backend_type;\n \n+void register_refs_backend(struct ref_be *be)\n+{\n+\tbe->next = refs_backends;\n+\trefs_backends = be;\n+}\n+\n /*\n * This function is used to switch to an alternate backend.\n */\ndiff --git a/refs.h b/refs.h\nindex cf1780e..ac88371 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -764,5 +764,6 @@ struct ref_be {\n \n extern struct ref_be refs_be_files;\n int set_refs_backend(const char *name, void *init_data);\n+void register_refs_backend(struct ref_be *be);\n \n #endif /* REFS_H */\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270860","messageId":"1443477738-32023-43-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 42/43] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:17Z","receivedAt":"2015-09-28T22:02:17Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a database backend for refs using LMDB. This backend runs git\nfor-each-ref about 30% faster than the files backend with fully-packed\nrefs on a repo with ~120k refs. It's also about 4x faster than using\nfully-unpacked refs. In addition, and perhaps more importantly , it\navoids case-conflict issues on OS X.\n\nLMDB has a few features that make it suitable for usage in git:\n\n1. It is relatively lightweight; it requires only one header file, and\nthe library code takes under 64k at runtime.\n\n2. It is well-tested: it's been used in OpenLDAP for years.\n\n3. It's very fast. LMDB's benchmarks show that it is among\nthe fastest key-value stores.\n\n4. It has a relatively simple concurrency story; readers don't\nblock writers and writers don't block readers.\n\nRonnie Sahlberg's original version of this patchset used tdb. The\nmajor disadvantage of tdb is that tdb is hard to build on OS X. It's\nalso not in homebrew. So lmdb seemed simpler.\n\nTo test this backend's correctness, I hacked test-lib.sh and\ntest-lib-functions.sh to run all tests under the refs backend. Dozens\nof tests use manual ref/reflog reading/writing, or create submodules\nwithout passing --refs-backend-type to git init. If those tests are\nchanged to use the update-ref machinery or test-refs-be-db (or, in the\ncase of packed-refs, corrupt refs, and dumb fetch tests, are skipped),\nthe only remaining failing tests are the git-new-workdir tests and the\ngitweb tests.\n\nSigned-off-by: David Turner \n---\n .gitignore | 1 +\n Documentation/git-clone.txt | 2 +-\n Documentation/git-init-db.txt | 2 +-\n Documentation/git-init.txt | 6 +\n Documentation/technical/refs-be-lmdb.txt | 39 +\n Makefile | 12 +\n builtin/init-db.c | 13 +\n config.c | 27 +\n configure.ac | 33 +\n contrib/workdir/git-new-workdir | 2 +\n environment.c | 1 +\n refs-be-lmdb.c | 2003 ++++++++++++++++++++++++++++++\n refs.h | 10 +\n setup.c | 28 +-\n test-refs-be-lmdb.c | 68 +\n 15 files changed, 2239 insertions(+), 8 deletions(-)\n create mode 100644 Documentation/technical/refs-be-lmdb.txt\n create mode 100644 refs-be-lmdb.c\n create mode 100644 test-refs-be-lmdb.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 1c2f832..7decc2f 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -199,6 +199,7 @@\n /test-path-utils\n /test-prio-queue\n /test-read-cache\n+/test-refs-be-lmdb\n /test-regex\n /test-revision-walking\n /test-run-command\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex d7a4cb0..6a4cf28 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -219,7 +219,7 @@ objects from the source repository into a pack in the cloned repository.\n \n --refs-backend-type=::\n \tType of refs backend. Default is to use the original files based\n-\tbackend.\n+\tbackend. Set to \"lmdb\" to activate the lmdb database backend.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt\nindex 648a6cd..72fbd71 100644\n--- a/Documentation/git-init-db.txt\n+++ b/Documentation/git-init-db.txt\n@@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n SYNOPSIS\n --------\n [verse]\n-'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]]\n+'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]] [--refs-backend-type=]\n \n \n DESCRIPTION\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex 8174d27..d352788 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -12,6 +12,7 @@ SYNOPSIS\n 'git init' [-q | --quiet] [--bare] [--template=]\n \t [--separate-git-dir ]\n \t [--shared[=]] [directory]\n+\t [--refs-backend-type=]\n \n \n DESCRIPTION\n@@ -113,6 +114,11 @@ does not exist, it will be created.\n \n --\n \n+--refs-backend-type=::\n+Type of refs backend. Default is to use the original \"files\" backend,\n+which stores ref data in files in .git/refs and .git/packed-refs. Set\n+to \"lmdb\" to activate the lmdb database backend.\n+\n TEMPLATE DIRECTORY\n ------------------\n \ndiff --git a/Documentation/technical/refs-be-lmdb.txt b/Documentation/technical/refs-be-lmdb.txt\nnew file mode 100644\nindex 0000000..98c13ec\n--- /dev/null\n+++ b/Documentation/technical/refs-be-lmdb.txt\n@@ -0,0 +1,39 @@\n+Notes on the LMDB refs backend\n+==============================\n+\n+Design:\n+------\n+\n+Refs and reflogs are stored in a lmdb database in .git/refdb. All\n+keys and values are \\0-terminated.\n+\n+Keys for refs are the name of the ref (e.g. refs/heads/master).\n+Values are the value of the ref\n+(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n+\n+All per-worktree refs (refs/bisect/* and HEAD) are store using\n+the traditional files-based backend.\n+\n+Reflogs are stored as a series of database entries.\n+\n+For an empty reflog, there is a \"header\" entry to show that a reflog\n+exists. For non-empty reflogs, there is one entry per logged ref\n+update. The key format is logs/[refname]\\0[timestamp]. The timestamp\n+is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n+This means that reflog entries are chronologically ordered. Because\n+LMDB is a btree database, we can efficiently iterate over these keys.\n+\n+Reflog values are in the same format as the original files-based\n+reflog.\n+\n+Weaknesses:\n+-----------\n+\n+The reflog format is somewhat inefficient: a binary format could store\n+reflog date/time information in somewhat less space.\n+\n+The rsync and file:// transports don't work yet, because they\n+don't use the refs API.\n+\n+git new-workdir is incompatible with the lmdb backend. Fortunately,\n+git new-workdir is deprecated, and worktrees work fine.\ndiff --git a/Makefile b/Makefile\nindex 43ceab0..e96f84b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1036,6 +1036,17 @@ ifdef USE_LIBPCRE\n \tEXTLIBS += -lpcre\n endif\n \n+ifdef USE_LIBLMDB\n+\tBASIC_CFLAGS += -DUSE_LIBLMDB\n+\tifdef LIBLMDBDIR\n+\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n+\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n+\tendif\n+\tEXTLIBS += -llmdb\n+\tLIB_OBJS += refs-be-lmdb.o\n+\tTEST_PROGRAMS_NEED_X += test-refs-be-lmdb\n+endif\n+\n ifdef HAVE_ALLOCA_H\n \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n endif\n@@ -2122,6 +2133,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n+\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 504a2dc..89f2c05 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -203,6 +203,17 @@ static int create_default_files(const char *template_path)\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\tif (refs_backend_type) {\n+\t\tstruct refdb_config_data config_data = {NULL};\n+\t\tgit_config_set(\"core.refs-backend-type\", refs_backend_type);\n+\t\tconfig_data.refs_backend_type = refs_backend_type;\n+\t\tconfig_data.refs_base = get_git_dir();\n+#ifdef USE_LIBLMDB\n+\t\tregister_refs_backend(&refs_be_lmdb);\n+#endif\n+\t\tset_refs_backend(refs_backend_type, &config_data);\n+\t}\n+\n \tif (refs_initdb(&err, shared_repository))\n \t\tdie(\"failed to set up refs db: %s\", err.buf);\n \n@@ -468,6 +479,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n+\t\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n+\t\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/config.c b/config.c\nindex 248a21a..9ce2ed2 100644\n--- a/config.c\n+++ b/config.c\n@@ -10,6 +10,7 @@\n #include \"exec_cmd.h\"\n #include \"strbuf.h\"\n #include \"quote.h\"\n+#include \"refs.h\"\n #include \"hashmap.h\"\n #include \"string-list.h\"\n #include \"utf8.h\"\n@@ -1207,6 +1208,32 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n \t}\n \n \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n+\t\tstruct refdb_config_data refdb_data = {NULL};\n+\t\tchar *repo_config_copy;\n+\n+\t\t/*\n+\t\t * make sure we always read the backend config from the\n+\t\t * core section on startup\n+\t\t */\n+\t\tret += git_config_from_file(refdb_config, repo_config,\n+\t\t\t\t\t &refdb_data);\n+\n+\t\trepo_config_copy = xstrdup(repo_config);\n+\t\trefdb_data.refs_base = strdup(dirname(repo_config_copy));\n+\t\tfree(repo_config_copy);\n+\n+\t\tif (refdb_data.refs_backend_type &&\n+\t\t !strcmp(refdb_data.refs_backend_type, \"lmdb\")) {\n+\n+#ifdef USE_LIBLMDB\n+\t\t\trefs_backend_type = refdb_data.refs_backend_type;\n+\t\t\tregister_refs_backend(&refs_be_lmdb);\n+\t\t\tset_refs_backend(refs_backend_type, &refdb_data);\n+#else\n+\t\t\tdie(\"Git was not built with USE_LIBLMDB, so the db refs backend is not available\");\n+#endif\n+\t\t}\n+\n \t\tret += git_config_from_file(fn, repo_config, data);\n \t\tfound += 1;\n \t}\ndiff --git a/configure.ac b/configure.ac\nindex 63b7244..ea8e93d 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -271,6 +271,24 @@ AS_HELP_STRING([], [ARG can be also prefix for libpcre library and hea\n dnl it yet.\n \tGIT_CONF_SUBST([LIBPCREDIR])\n fi)\n+\n+USE_LIBLMDB=YesPlease\n+AC_ARG_WITH(liblmdb,\n+AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])\n+AS_HELP_STRING([], [ARG can be also prefix for liblmdb library and headers]),\n+ if test \"$withval\" = \"no\"; then\n+\tUSE_LIBLMDB=\n+ elif test \"$withval\" = \"yes\"; then\n+\tUSE_LIBLMDB=YesPlease\n+ else\n+\tUSE_LIBLMDB=YesPlease\n+\tLIBLMDBDIR=$withval\n+\tAC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])\n+ dnl USE_LIBLMDB can still be modified below, so don't substitute\n+ dnl it yet.\n+\tGIT_CONF_SUBST([LIBLMDBDIR])\n+ fi)\n+\n #\n # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n AC_FUNC_ALLOCA\n@@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])\n \n fi\n \n+if test -n \"$USE_LIBLMDB\"; then\n+\n+GIT_STASH_FLAGS($LIBLMDBDIR)\n+\n+AC_CHECK_LIB([lmdb], [mdb_env_open],\n+[USE_LIBLMDB=YesPlease],\n+[USE_LIBLMDB=])\n+\n+GIT_UNSTASH_FLAGS($LIBLMDBDIR)\n+\n+GIT_CONF_SUBST([USE_LIBLMDB])\n+\n+fi\n+\n+\n #\n # Define NO_CURL if you do not have libcurl installed. git-http-pull and\n # git-http-push are not built, and you cannot use http:// and https://\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a..9761a1a 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -28,6 +28,8 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n git rev-parse --git-dir 2>/dev/null) ||\n die \"Not a git repository: \\\"$orig_git\\\"\"\n \n+test -d $git_dir/refdb && die \"git-new-workdir is incompatible with the refs db backend\"\n+\n case \"$git_dir\" in\n .git)\n \tgit_dir=\"$orig_git/.git\"\ndiff --git a/environment.c b/environment.c\nindex c5b65f5..6a2f639 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -65,6 +65,7 @@ int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+const char *refs_backend_type;\n \n #ifndef PROTECT_HFS_DEFAULT\n #define PROTECT_HFS_DEFAULT 0\ndiff --git a/refs-be-lmdb.c b/refs-be-lmdb.c\nnew file mode 100644\nindex 0000000..99cbd29\n--- /dev/null\n+++ b/refs-be-lmdb.c\n@@ -0,0 +1,2003 @@\n+/*\n+ * This file implements a lmdb backend for refs.\n+ *\n+ * The design of this backend relies on lmdb's write lock -- that is, any\n+ * write transaction blocks all other writers. Thus, as soon as a ref\n+ * transaction is opened, we know that any values we read won't\n+ * change out from under us, and we have a fully-consistent view of the\n+ * database.\n+ *\n+ * We store the content of refs including the trailing \\0 so that\n+ * standard C string functions can handle them. Just like struct\n+ * strbuf.\n+ */\n+#include \n+#include \n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"refs.h\"\n+#include \"tag.h\"\n+#include \"lockfile.h\"\n+#include \"run-command.h\"\n+\n+static struct trace_key db_trace = TRACE_KEY_INIT(LMDB);\n+\n+static MDB_env *env;\n+\n+static char *db_path;\n+\n+struct lmdb_transaction_info {\n+\tMDB_txn *txn;\n+\tMDB_dbi dbi;\n+\tconst char *submodule;\n+};\n+\n+struct lmdb_transaction {\n+\tstruct ref_transaction base;\n+\tMDB_cursor *cursor;\n+\tstruct lmdb_transaction_info info;\n+\tstruct hashmap updated_refs;\n+\tunsigned int flags;\n+};\n+\n+static struct lmdb_transaction transaction = {{}, NULL};\n+\n+struct ref_update {\n+\tstruct hashmap_entry ent; /* must be first */\n+\tsize_t len;\n+\tchar refname[FLEX_ARRAY];\n+};\n+\n+static char *get_refdb_path(const char *base)\n+{\n+\tstruct strbuf path_buf = STRBUF_INIT;\n+\tstrbuf_addf(&path_buf, \"%s/refdb\", base);\n+\treturn strbuf_detach(&path_buf, NULL);\n+}\n+\n+static int in_write_transaction(void)\n+{\n+\treturn transaction.info.txn && !(transaction.flags & MDB_RDONLY);\n+}\n+\n+static void init_env(MDB_env **env, const char *path)\n+{\n+\tint ret;\n+\tif (*env)\n+\t\treturn;\n+\n+\tif ((ret = mdb_env_create(env)) != MDB_SUCCESS)\n+\t\tdie(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n+\tif ((ret = mdb_env_set_maxreaders(*env, 1000)) != MDB_SUCCESS)\n+\t\tdie(\"BUG: mdb_env_set_maxreaders failed: %s\", mdb_strerror(ret));\n+\tif ((ret = mdb_env_set_mapsize(*env, (1<<30))) != MDB_SUCCESS)\n+\t\tdie(\"BUG: mdb_set_mapsize failed: %s\", mdb_strerror(ret));\n+\tif ((ret = mdb_env_open(*env, path, 0 , 0664)) != MDB_SUCCESS)\n+\t\tdie(\"BUG: mdb_env_open (%s) failed: %s\", path, mdb_strerror(ret));\n+}\n+\n+static int lmdb_initdb(struct strbuf *err, int shared)\n+{\n+\t/* To create a db, all we need to do is make a directory for\n+\t it to live in; lmdb will do the rest. */\n+\n+\tassert(db_path);\n+\tif (mkdir(db_path, 0775)) {\n+\t\tif (errno != EEXIST) {\n+\t\t\tstrbuf_addf(err, \"%s\", strerror(errno));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static void lmdb_init_backend(struct refdb_config_data *data)\n+{\n+\tif (db_path)\n+\t\treturn;\n+\n+\tdb_path = xstrdup(real_path(get_refdb_path(data->refs_base)));\n+\ttrace_printf_key(&db_trace, \"Init backend\\n\");\n+}\n+\n+int ref_update_cmp(const void *entry, const void *entry_or_key, const void *keydata)\n+{\n+\n+\tconst struct ref_update *existing = entry;\n+\tconst struct ref_update *incoming = entry_or_key;\n+\n+\treturn existing->len != incoming->len ||\n+\t\tmemcmp(existing->refname, incoming->refname, existing->len);\n+}\n+\n+static void mdb_cursor_open_or_die(struct lmdb_transaction_info *info,\n+\t\t\t\t MDB_cursor **cursor)\n+{\n+\tint ret = mdb_cursor_open(info->txn, info->dbi, cursor);\n+\tif (ret)\n+\t\tdie(\"mdb_cursor_open failed: %s\", mdb_strerror(ret));\n+}\n+\n+static void submodule_path(struct strbuf *sb, const char *submodule,\n+\t\t\t const char *refname)\n+{\n+\tif (submodule)\n+\t\tstrbuf_git_path_submodule(sb, submodule, \"%s\", refname);\n+\telse\n+\t\tstrbuf_git_path(sb, \"%s\", refname);\n+}\n+\n+static int read_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t struct MDB_val *val)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+\tif (strbuf_read_file(&sb, path.buf, 200) < 0) {\n+\t\tstrbuf_release(&sb);\n+\t\tstrbuf_release(&path);\n+\t\tif (errno == ENOENT)\n+\t\t\treturn MDB_NOTFOUND;\n+\t\treturn -1;\n+\t}\n+\tstrbuf_setlen(&sb, sb.len - 1);\n+\n+\tval->mv_data = sb.buf;\n+\tval->mv_size = sb.len + 1;\n+\n+\tstrbuf_detach(&sb, NULL);\n+\tstrbuf_release(&path);\n+\treturn 0;\n+}\n+\n+static void write_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\t MDB_val *val)\n+{\n+\tstatic struct lock_file lock;\n+\tint fd;\n+\tint len = val->mv_size - 1;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\tsafe_create_leading_directories(path.buf);\n+\n+\tfd = hold_lock_file_for_update(&lock, path.buf, LOCK_DIE_ON_ERROR);\n+\tstrbuf_release(&path);\n+\n+\tif (write_in_full(fd, val->mv_data, len) != len ||\n+\t write_in_full(fd, \"\\n\", 1) != 1)\n+\t\tdie_errno(\"failed to write new HEAD\");\n+\n+\tif (commit_lock_file(&lock))\n+\t\tdie_errno(\"failed to write new HEAD\");\n+}\n+\n+static int del_per_worktree_ref(const char *submodule, const char *refname,\n+\t\t\t\tMDB_val *val)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint result;\n+\n+\t/*\n+\t * Returning deleted ref data is not yet implemented, but no\n+\t * callers need it.\n+\t */\n+\tassert(val == NULL);\n+\n+\tsubmodule_path(&path, submodule, refname);\n+\n+\tresult = unlink(path.buf);\n+\tstrbuf_release(&path);\n+\tif (result) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int mdb_get_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\treturn read_per_worktree_ref(info->submodule, key->mv_data, val);\n+\n+\tret = mdb_get(info->txn, info->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static int mdb_del_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n+\t\treturn del_per_worktree_ref(info->submodule, key->mv_data, val);\n+\n+\tret = mdb_del(info->txn, info->dbi, key, val);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static void mdb_put_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL) {\n+\t\twrite_per_worktree_ref(info->submodule, key->mv_data, val);\n+\t\treturn;\n+\t}\n+\tassert(val->mv_size == 0 || ((char *)val->mv_data)[val->mv_size - 1] == 0);\n+\n+\tret = mdb_put(info->txn, info->dbi, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret == MDB_BAD_VALSIZE)\n+\t\t\tdie(\"Ref name %s too long (max size is %d)\",\n+\t\t\t (const char *)key->mv_data,\n+\t\t\t mdb_env_get_maxkeysize(env));\n+\t\telse\n+\t\t\tdie(\"mdb_put failed: %s\", mdb_strerror(ret));\n+\t}\n+}\n+\n+static int mdb_cursor_get_or_die(MDB_cursor *cursor, MDB_val *key, MDB_val *val, int mode)\n+{\n+\tint ret;\n+\n+\tret = mdb_cursor_get(cursor, key, val, mode);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_cursor_get failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\tassert(((char *)val->mv_data)[val->mv_size - 1] == 0);\n+\treturn 0;\n+}\n+\n+static int mdb_cursor_del_or_die(MDB_cursor *cursor, int flags)\n+{\n+\tint ret = mdb_cursor_del(cursor, flags);\n+\tif (ret) {\n+\t\tif (ret != MDB_NOTFOUND)\n+\t\t\tdie(\"mdb_cursor_del failed: %s\", mdb_strerror(ret));\n+\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * Begin a transaction. Because only one transaction per thread is\n+ * permitted, we use a global transaction object. If a read-write\n+ * transaction is presently already in-progress, and a read-only\n+ * transaction is requested, the read-write transaction will be\n+ * returned instead. If a read-write transaction is requested and a\n+ * read-only transaction is open, the read-only transaction will be\n+ * closed.\n+ *\n+ * It is a bug to request a read-write transaction during another\n+ * read-write transaction.\n+ *\n+ * As a result, it is unsafe to retain read-only transactions past the\n+ * point where a read-write transaction might be needed. For\n+ * instance, any call that has callbacks outside this module must\n+ * conclude all of its reads from the database before calling those\n+ * callbacks, or must reacquire the transaction after its callbacks\n+ * are completed.\n+ */\n+int lmdb_transaction_begin_flags(struct strbuf *err, unsigned int flags)\n+{\n+\tint ret;\n+\tMDB_txn *txn;\n+\tstatic int last_commands_run = 0;\n+\tint force_restart = 0;\n+\n+\tinit_env(&env, db_path);\n+\n+\tif (total_commands_run != last_commands_run) {\n+\t\t/*\n+\t\t * Since each transaction sees a consistent view of\n+\t\t * the db, downstream processes that write the db\n+\t\t * won't be seen in this transaction. We don't know\n+\t\t * whether any given downstream process has made any\n+\t\t * writes, so if there have been any downstream processes,\n+\t\t * we had better reopen the transaction.\n+\t\t */\n+\t\tforce_restart = 1;\n+\t\tlast_commands_run = total_commands_run;\n+\t}\n+\n+\tif (!transaction.info.txn) {\n+\t\thashmap_init(&transaction.updated_refs, ref_update_cmp, 0);\n+\t\tif ((ret = mdb_txn_begin(env, NULL, flags, &txn)) != MDB_SUCCESS) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif ((ret = mdb_dbi_open(txn, NULL, 0, &transaction.info.dbi)) != MDB_SUCCESS) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.info.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t\treturn 0;\n+\t}\n+\n+\tif (transaction.flags == flags && !(flags & MDB_RDONLY)) {\n+\t\tdie(\"BUG: rw transaction started during another rw txn\");\n+\t}\n+\n+\tif (force_restart || (transaction.flags != flags && transaction.flags & MDB_RDONLY)) {\n+\t\t/*\n+\t\t * RO -> RW, or forced restart due to possible changes\n+\t\t * from downstream processes.\n+\t\t */\n+\t\tmdb_txn_abort(transaction.info.txn);\n+\t\tif ((ret = mdb_txn_begin(env, NULL, flags, &txn)) != MDB_SUCCESS) {\n+\t\t\tstrbuf_addf(err, \"restarting txn: mdb_txn_begin failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif ((ret = mdb_dbi_open(txn, NULL, 0, &transaction.info.dbi)) != MDB_SUCCESS) {\n+\t\t\tstrbuf_addf(err, \"mdb_txn_open failed: %s\",\n+\t\t\t\t mdb_strerror(ret));\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttransaction.info.txn = txn;\n+\t\ttransaction.flags = flags;\n+\t}\n+\t/* RW -> RO just keeps the RW txn */\n+\treturn 0;\n+}\n+\n+static struct ref_transaction *lmdb_transaction_begin_flags_or_die(int flags)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tif (lmdb_transaction_begin_flags(&err, flags))\n+\t\tdie(\"%s\", err.buf);\n+\treturn (struct ref_transaction *)&transaction;\n+}\n+\n+static struct ref_transaction *lmdb_transaction_begin(struct strbuf *err)\n+{\n+\tif (lmdb_transaction_begin_flags(err, 0))\n+\t\treturn NULL;\n+\treturn (struct ref_transaction *)&transaction;\n+}\n+\n+#define MAXDEPTH 5\n+\n+static const char *parse_ref_data(struct lmdb_transaction_info *info,\n+\t\t\t\t const char *refname, const char *ref_data,\n+\t\t\t\t unsigned char *sha1, int resolve_flags,\n+\t\t\t\t int *flags, int bad_name)\n+{\n+\tint depth = MAXDEPTH;\n+\tconst char *buf;\n+\tstatic struct strbuf refname_buffer = STRBUF_INIT;\n+\tstatic struct strbuf refdata_buffer = STRBUF_INIT;\n+\tMDB_val key, val;\n+\tconst char *ret;\n+\n+\tstrbuf_reset(&refname_buffer);\n+\tstrbuf_reset(&refdata_buffer);\n+\n+\tfor (;;) {\n+\t\tif (--depth < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tif (!starts_with(ref_data, \"ref:\")) {\n+\t\t\tif (get_sha1_hex(ref_data, sha1) ||\n+\t\t\t (ref_data[40] != '\\0' && !isspace(ref_data[40]))) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tif (bad_name) {\n+\t\t\t\thashclr(sha1);\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t} else if (is_null_sha1(sha1)) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (flags)\n+\t\t\t*flags |= REF_ISSYMREF;\n+\t\tbuf = ref_data + 4;\n+\t\twhile (isspace(*buf))\n+\t\t\tbuf++;\n+\t\tstrbuf_addf(&refname_buffer, buf);\n+\t\trefname = refname_buffer.buf;\n+\t\tif (resolve_flags & RESOLVE_REF_NO_RECURSE) {\n+\t\t\thashclr(sha1);\n+\t\t\treturn refname;\n+\t\t}\n+\t\tif (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\n+\t\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t\t !refname_is_safe(buf)) {\n+\t\t\t\terrno = EINVAL;\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\tbad_name = 1;\n+\t\t}\n+\n+\t\tkey.mv_data = (char *)refname;\n+\t\tkey.mv_size = strlen(refname) + 1;\n+\t\tif (mdb_get_or_die(info, &key, &val)) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (bad_name) {\n+\t\t\t\tif (flags)\n+\t\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\t\treturn NULL;\n+\n+\t\t\treturn refname;\n+\t\t}\n+\t\tstrbuf_reset(&refdata_buffer);\n+\t\tstrbuf_add(&refdata_buffer, val.mv_data, val.mv_size);\n+\t\tref_data = refdata_buffer.buf;\n+\t}\n+\treturn refname;\n+}\n+\n+static int verify_refname_available_txn(struct lmdb_transaction_info *info,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tstruct string_list *extras,\n+\t\t\t\t\tstruct string_list *skip,\n+\t\t\t\t\tstruct strbuf *err)\n+{\n+\tMDB_cursor *cursor;\n+\tMDB_val key;\n+\tMDB_val val;\n+\tint mdb_ret;\n+\tsize_t refname_len;\n+\tchar *search_key;\n+\tconst char *extra_refname;\n+\tint ret = 1;\n+\tsize_t i;\n+\n+\tmdb_cursor_open_or_die(info, &cursor);\n+\n+\trefname_len = strlen(refname) + 2;\n+\tkey.mv_size = refname_len;\n+\tsearch_key = xmalloc(refname_len);\n+\tmemcpy(search_key, refname, refname_len - 2);\n+\tsearch_key[refname_len - 2] = '/';\n+\tsearch_key[refname_len - 1] = 0;\n+\tkey.mv_data = search_key;\n+\n+\t/*\n+\t * Check for subdirs of refname: we start at refname/\n+\t */\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (starts_with(key.mv_data, refname) &&\n+\t\t ((char*)key.mv_data)[refname_len - 2] == '/') {\n+\t\t\tif (skip && string_list_has_string(skip, key.mv_data))\n+\t\t\t\tgoto next;\n+\n+\t\t\tstrbuf_addf(err, \"'%s' exists; cannot create '%s'\", (char *)key.mv_data, refname);\n+\t\t\tgoto done;\n+\t\t}\n+\t\tbreak;\n+\tnext:\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\t/* Check for parent dirs of refname. */\n+\tfor (i = 0; i < refname_len - 2; ++i) {\n+\t\tif (search_key[i] == '/') {\n+\t\t\tsearch_key[i] = 0;\n+\t\t\tif (skip && string_list_has_string(skip, search_key)) {\n+\t\t\t\tsearch_key[i] = '/';\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, search_key)) {\n+\t\t\t\tstrbuf_addf(err, \"cannot process '%s' and '%s' at the same time\",\n+\t\t\t\t\t refname, search_key);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tkey.mv_data = search_key;\n+\t\t\tkey.mv_size = i + 1;\n+\t\t\tif (!mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET)) {\n+\t\t\t\tstrbuf_addf(err, \"'%s' exists; cannot create '%s'\", (char *)key.mv_data, refname);\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tsearch_key[i] = '/';\n+\t\t}\n+\t}\n+\n+\textra_refname = find_descendant_ref(refname, extras, skip);\n+\tif (extra_refname) {\n+\t\tstrbuf_addf(err,\n+\t\t\t \"cannot process '%s' and '%s' at the same time\",\n+\t\t\t refname, extra_refname);\n+\t\tret = 1;\n+\t} else {\n+\t\tret = 0;\n+\t}\n+done:\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\treturn ret;\n+}\n+\n+static const char *resolve_ref_unsafe_txn(struct lmdb_transaction_info *info,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1,\n+\t\t\t\t\t int *flags)\n+{\n+\tint bad_name = 0;\n+\tchar *ref_data;\n+\tstruct MDB_val key, val;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\n+\tif (flags)\n+\t\t*flags = 0;\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (flags)\n+\t\t\t*flags |= REF_BAD_NAME;\n+\n+\t\tif (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||\n+\t\t !refname_is_safe(refname)) {\n+\t\t\terrno = EINVAL;\n+\t\t\treturn NULL;\n+\t\t}\n+\t\t/*\n+\t\t * dwim_ref() uses REF_ISBROKEN to distinguish between\n+\t\t * missing refs and refs that were present but invalid,\n+\t\t * to complain about the latter to stderr.\n+\t\t *\n+\t\t * We don't know whether the ref exists, so don't set\n+\t\t * REF_ISBROKEN yet.\n+\t\t */\n+\t\tbad_name = 1;\n+\t}\n+\n+\tkey.mv_data = (void *)refname;\n+\tkey.mv_size = strlen(refname) + 1;\n+\tif (mdb_get_or_die(info, &key, &val)) {\n+\t\tif (bad_name) {\n+\t\t\thashclr(sha1);\n+\t\t\tif (flags)\n+\t\t\t\t*flags |= REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (resolve_flags & RESOLVE_REF_READING)\n+\t\t\treturn NULL;\n+\n+\t\tif (verify_refname_available_txn(info, refname, NULL, NULL, &err)) {\n+\t\t\terror(\"%s\", err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\thashclr(sha1);\n+\t\treturn refname;\n+\t}\n+\n+\tref_data = val.mv_data;\n+\tassert(ref_data[val.mv_size - 1] == 0);\n+\n+\treturn parse_ref_data(info, refname, ref_data, sha1, resolve_flags,\n+\t\t\t flags, bad_name);\n+}\n+\n+static const char *lmdb_resolve_ref_unsafe(const char *refname, int resolve_flags,\n+\t\t\t\t\t unsigned char *sha1, int *flags)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn resolve_ref_unsafe_txn(&transaction.info, refname,\n+\t\t\t\t resolve_flags, sha1, flags);\n+}\n+\n+static void write_u64(char *buf, uint64_t number)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < 8; ++i)\n+\t\tbuf[i] = (number >> (i * 8)) & 0xff;\n+}\n+\n+static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tchar *email_end, *message;\n+\tunsigned long timestamp;\n+\tint tz;\n+\n+\t/* old (raw) new (raw) name SP time TAB msg LF */\n+\tif (sb->len < 41 || sb->buf[sb->len - 1] != '\\n' ||\n+\t !(email_end = strchr(sb->buf + 40, '>')) ||\n+\t email_end[1] != ' ' ||\n+\t !(timestamp = strtoul(email_end + 2, &message, 10)) ||\n+\t !message || message[0] != ' ' ||\n+\t (message[1] != '+' && message[1] != '-') ||\n+\t !isdigit(message[2]) || !isdigit(message[3]) ||\n+\t !isdigit(message[4]) || !isdigit(message[5]))\n+\t\treturn 0; /* corrupt? */\n+\n+\thashcpy(osha1, (const unsigned char *)sb->buf);\n+\thashcpy(nsha1, (const unsigned char *)sb->buf + 20);\n+\n+\temail_end[1] = '\\0';\n+\ttz = strtol(message + 1, NULL, 10);\n+\tif (message[6] != '\\t')\n+\t\tmessage += 6;\n+\telse\n+\t\tmessage += 7;\n+\treturn fn(osha1, nsha1, sb->buf + 40, timestamp, tz, message, cb_data);\n+}\n+\n+static void format_reflog_entry(struct strbuf *buf,\n+\t\t\t\tconst unsigned char *old_sha1,\n+\t\t\t\tconst unsigned char *new_sha1,\n+\t\t\t\tconst char *committer, const char *msg)\n+{\n+\tint len;\n+\tint msglen;\n+\n+\tassert(buf->len == 0);\n+\tstrbuf_add(buf, old_sha1, 20);\n+\tstrbuf_add(buf, new_sha1, 20);\n+\tstrbuf_addstr(buf, committer);\n+\tstrbuf_addch(buf, '\\n');\n+\n+\tlen = buf->len;\n+\tmsglen = msg ? strlen(msg) : 0;\n+\tif (msglen) {\n+\t\tstrbuf_grow(buf, msglen + 1);\n+\t\tint copied = copy_reflog_msg(buf->buf + 40 + strlen(committer), msg) - 1;\n+\t\tbuf->len = len + copied;\n+\t\tbuf->buf[buf->len] = 0;\n+\t}\n+}\n+\n+static int log_ref_write(const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t const char *msg,\n+\t\t\t int flags,\n+\t\t\t struct strbuf *err)\n+{\n+\tMDB_val key, val;\n+\tuint64_t now = getnanotime();\n+\tint result;\n+\tchar *log_key;\n+\tint refname_len;\n+\tMDB_cursor *cursor;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *timestamp;\n+\n+\tif (log_all_ref_updates < 0)\n+\t\tlog_all_ref_updates = !is_bare_repository();\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL) {\n+\t\treturn files_log_ref_write(refname, old_sha1, new_sha1,\n+\t\t\t\t\t msg, flags, err);\n+\t}\n+\n+\t/* it is assumed that we are in a ref transaction here */\n+\tassert(transaction.info.txn);\n+\n+\tresult = safe_create_reflog(refname, flags & REF_FORCE_CREATE_REFLOG, err);\n+\tif (result)\n+\t\treturn result;\n+\n+\t/* \"logs/\" + refname + \\0 + 8-byte timestamp for sorting and expiry. */\n+\trefname_len = strlen(refname);\n+\tkey.mv_size = refname_len + 14;\n+\tlog_key = xcalloc(1, key.mv_size);\n+\tsprintf(log_key, \"logs/%s\", refname);\n+\tkey.mv_data = log_key;\n+\n+\tmdb_cursor_open_or_die(&transaction.info, &cursor);\n+\n+\t/* check that a reflog exists */\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||\n+\t strcmp(key.mv_data, log_key)) {\n+\t\tfree(log_key);\n+\t\tmdb_cursor_close(cursor);\n+\t\treturn 0;\n+\t}\n+\n+\t/* Is this a header? We only need the header for empty reflogs */\n+\ttimestamp = (const char *)key.mv_data + refname_len + 6;\n+\tif (ntohll(*(uint64_t *)timestamp) == 0)\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\n+\tkey.mv_data = log_key;\n+\n+\twrite_u64(key.mv_data + refname_len + 6, htonll(now));\n+\n+\tformat_reflog_entry(&buf, old_sha1, new_sha1,\n+\t\t\t git_committer_info(0), msg);\n+\tassert(buf.len >= 42);\n+\tval.mv_data = buf.buf;\n+\tval.mv_size = buf.len + 1;\n+\n+\tmdb_put_or_die(&transaction.info, &key, &val, 0);\n+\n+\tstrbuf_release(&buf);\n+\tfree(log_key);\n+\tmdb_cursor_close(cursor);\n+\treturn 0;\n+}\n+\n+static int lmdb_verify_refname_available(const char *refname,\n+\t\t\t\t\t struct string_list *extras,\n+\t\t\t\t\t struct string_list *skip,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn verify_refname_available_txn(&transaction.info, refname, extras, skip, err);\n+}\n+\n+static const char *check_ref(MDB_txn *txn, const char *refname,\n+\t\t\t const unsigned char *old_sha1,\n+\t\t\t unsigned char *resolved_sha1, int flags,\n+\t\t\t int *type_p)\n+{\n+\tint mustexist = (old_sha1 && !is_null_sha1(old_sha1));\n+\tint resolve_flags = 0;\n+\tint type;\n+\n+\tif (mustexist)\n+\t\tresolve_flags |= RESOLVE_REF_READING;\n+\tif (flags & REF_DELETING) {\n+\t\tresolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;\n+\t\tif (flags & REF_NODEREF)\n+\t\t\tresolve_flags |= RESOLVE_REF_NO_RECURSE;\n+\t}\n+\n+\trefname = resolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t resolved_sha1, &type);\n+\tif (type_p)\n+\t *type_p = type;\n+\tif (!refname) {\n+\t\treturn NULL;\n+\t}\n+\n+\tif (old_sha1) {\n+\t\tif (flags & REF_NODEREF) {\n+\t\t\tresolve_flags &= ~RESOLVE_REF_NO_RECURSE;\n+\t\t\tresolve_ref_unsafe(refname, resolve_flags,\n+\t\t\t\t\t resolved_sha1, &type);\n+\t\t}\n+\t\tif (hashcmp(old_sha1, resolved_sha1)) {\n+\t\t\terror(\"Ref %s is at %s but expected %s\", refname,\n+\t\t\t sha1_to_hex(resolved_sha1), sha1_to_hex(old_sha1));\n+\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\treturn refname;\n+}\n+\n+int lmdb_transaction_create(struct ref_transaction *transaction,\n+\t\t\t const char *refname,\n+\t\t\t const unsigned char *new_sha1,\n+\t\t\t unsigned int flags, const char *msg,\n+\t\t\t struct strbuf *err)\n+{\n+\tif (!new_sha1 || is_null_sha1(new_sha1))\n+\t\tdie(\"BUG: create called without valid new_sha1\");\n+\treturn ref_transaction_update(transaction, refname, new_sha1,\n+\t\t\t\t null_sha1, flags, msg, err);\n+}\n+\n+static int lmdb_transaction_delete(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n+{\n+\t\tif (old_sha1 && is_null_sha1(old_sha1))\n+\t\tdie(\"BUG: delete called with old_sha1 set to zeros\");\n+\treturn ref_transaction_update(transaction, refname,\n+\t\t\t\t null_sha1, old_sha1,\n+\t\t\t\t flags, msg, err);\n+}\n+\n+static int lmdb_transaction_verify(struct ref_transaction *transaction,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags,\n+\t\t\t\t struct strbuf *err)\n+{\n+\t\tif (!old_sha1)\n+\t\tdie(\"BUG: verify called with old_sha1 set to NULL\");\n+\treturn ref_transaction_update(transaction, refname,\n+\t\t\t\t NULL, old_sha1,\n+\t\t\t\t flags, NULL, err);\n+}\n+\n+static void lmdb_transaction_free_1(struct lmdb_transaction *transaction)\n+{\n+\thashmap_free(&transaction->updated_refs, 1);\n+\ttransaction->info.txn = NULL;\n+}\n+\n+static void lmdb_transaction_free(struct ref_transaction *trans)\n+{\n+\tstruct lmdb_transaction *transaction = (struct lmdb_transaction *)trans;\n+\tif (!transaction->info.txn)\n+\t\treturn;\n+\n+\tmdb_txn_abort(transaction->info.txn);\n+\tlmdb_transaction_free_1(transaction);\n+\treturn;\n+}\n+\n+static int lmdb_transaction_commit(struct ref_transaction *trans,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct lmdb_transaction *transaction = (struct lmdb_transaction *)trans;\n+\tint result;\n+\n+\tresult = mdb_txn_commit(transaction->info.txn);\n+\tlmdb_transaction_free_1(transaction);\n+\treturn result;\n+}\n+\n+static int lmdb_delete_reflog(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tint mdb_ret;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tint in_transaction;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn files_delete_reflog(refname);\n+\n+\tin_transaction = in_write_transaction();\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction.info, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) || ((char*)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\ttransaction.cursor = NULL;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_commit((struct ref_transaction *)&transaction, &err);\n+\tstrbuf_release(&err);\n+\treturn ret;\n+}\n+\n+#define REF_NO_REFLOG 0x8000\n+\n+static int lmdb_transaction_update(struct ref_transaction *trans,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const unsigned char *new_sha1,\n+\t\t\t\t const unsigned char *old_sha1,\n+\t\t\t\t unsigned int flags, const char *msg,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct lmdb_transaction *transaction = (struct lmdb_transaction *)trans;\n+\tconst char *orig_refname = refname;\n+\tMDB_val key, val;\n+\tstruct ref_update *update, *old;\n+\tunsigned char resolved_sha1[20];\n+\tsize_t len;\n+\tint type;\n+\n+\tif (new_sha1)\n+\t\tflags |= REF_HAVE_NEW;\n+\tif (old_sha1)\n+\t\tflags |= REF_HAVE_OLD;\n+\n+\tif ((flags & REF_HAVE_NEW) && is_null_sha1(new_sha1))\n+\t\tflags |= REF_DELETING;\n+\n+\tif (new_sha1 && !is_null_sha1(new_sha1) &&\n+\t check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(err, \"refusing to update ref with bad name %s\",\n+\t\t\t refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\tlen = strlen(orig_refname);\n+\tupdate = xmalloc(sizeof(*update) + len + 1);\n+\tupdate->len = len;\n+\tstrcpy(update->refname, orig_refname);\n+\thashmap_entry_init(update, memhash(update->refname, len));\n+\told = hashmap_put(&(transaction->updated_refs), update);\n+\tif (old) {\n+\t\tstrbuf_addf(err, \"Multiple updates for ref '%s' not allowed.\",\n+\t\t\t orig_refname);\n+\t\tfree(old);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\trefname = check_ref(transaction->info.txn, orig_refname, old_sha1,\n+\t\t\t resolved_sha1, flags, &type);\n+\n+\tif (refname == NULL) {\n+\t\tstrbuf_addf(err, \"cannot lock the ref '%s'\", orig_refname);\n+\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t}\n+\n+\tif (!(flags & REF_DELETING) && is_null_sha1(resolved_sha1) &&\n+\t verify_refname_available_txn(&transaction->info, refname, NULL, NULL, err))\n+\t\treturn TRANSACTION_NAME_CONFLICT;\n+\n+\tif (flags & REF_NODEREF)\n+\t\trefname = orig_refname;\n+\n+\tkey.mv_size = strlen(refname) + 1;\n+\tkey.mv_data = (void *)refname;\n+\n+\tif ((flags & REF_HAVE_NEW) && !is_null_sha1(new_sha1)) {\n+\t\tint overwriting_symref = ((type & REF_ISSYMREF) &&\n+\t\t\t\t\t (flags & REF_NODEREF));\n+\n+\t\tstruct object *o = parse_object(new_sha1);\n+\t\tif (!o) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Trying to write ref %s with nonexistent object %s\",\n+\t\t\t\t refname, sha1_to_hex(new_sha1));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (o->type != OBJ_COMMIT && is_branch(refname)) {\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t \"Trying to write non-commit object %s to branch %s\",\n+\t\t\t\t sha1_to_hex(new_sha1), refname);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (!overwriting_symref\n+\t\t && !hashcmp(resolved_sha1, new_sha1)) {\n+\t\t\t/*\n+\t\t\t * The reference already has the desired\n+\t\t\t * value, so we don't need to write it.\n+\t\t\t */\n+\t\t\tflags |= REF_NO_REFLOG;\n+\t\t} else {\n+\t\t\tval.mv_size = 41;\n+\t\t\tif (new_sha1)\n+\t\t\t\tval.mv_data = sha1_to_hex(new_sha1);\n+\t\t\telse\n+\t\t\t\tval.mv_data = sha1_to_hex(null_sha1);\n+\t\t\tmdb_put_or_die(&transaction->info, &key, &val, 0);\n+\t\t}\n+\t}\n+\n+\tif (flags & REF_DELETING) {\n+\t\tif (mdb_del_or_die(&transaction->info, &key, NULL)) {\n+\t\t\tif (old_sha1 && !is_null_sha1(old_sha1)) {\n+\t\t\t\tstrbuf_addf(err, \"No such ref %s\", refname);\n+\t\t\t\treturn TRANSACTION_GENERIC_ERROR;\n+\t\t\t}\n+\t\t}\n+\t\tlmdb_delete_reflog(orig_refname);\n+\t} else if (!(flags & REF_NO_REFLOG)) {\n+\t\tif (log_ref_write(orig_refname, resolved_sha1,\n+\t\t\t\t new_sha1 ? new_sha1 : null_sha1,\n+\t\t\t\t msg, flags, err) < 0)\n+\t\t\treturn -1;\n+\t\tif (strcmp (refname, orig_refname) &&\n+\t\t log_ref_write(refname, resolved_sha1,\n+\t\t\t\t new_sha1 ? new_sha1 : null_sha1,\n+\t\t\t\t msg, flags, err) < 0)\n+\t\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int rename_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\n+\tconst char *newrefname = cb_data;\n+\tMDB_val key, new_key, val;\n+\n+\tassert(transaction.cursor);\n+\n+\tif (mdb_cursor_get_or_die(transaction.cursor, &key, &val, MDB_GET_CURRENT))\n+\t\tdie(\"renaming ref: mdb_cursor_get failed to get current\");\n+\n+\tnew_key.mv_size = strlen(newrefname) + 5 + 1 + 8;\n+\tnew_key.mv_data = xmalloc(new_key.mv_size);\n+\tstrcpy(new_key.mv_data, \"logs/\");\n+\tstrcpy(new_key.mv_data + 5, newrefname);\n+\tmemcpy(new_key.mv_data + new_key.mv_size - 8, key.mv_data + key.mv_size - 8, 8);\n+\tmdb_put_or_die(&transaction.info, &new_key, &val, 0);\n+\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\tfree(new_key.mv_data);\n+\treturn 0;\n+}\n+\n+static int lmdb_rename_ref(const char *oldref, const char *newref, const char *logmsg)\n+{\n+\tunsigned char orig_sha1[20];\n+\tint flag = 0;\n+\tint log = reflog_exists(oldref);\n+\tconst char *symref = NULL;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *ref_transaction;\n+\n+\tassert(!in_write_transaction());\n+\n+\tif (!strcmp(oldref, newref))\n+\t\treturn 0;\n+\n+\tref_transaction = lmdb_transaction_begin_flags_or_die(0);\n+\n+\tsymref = resolve_ref_unsafe(oldref, RESOLVE_REF_READING,\n+\t\t\t\t orig_sha1, &flag);\n+\tif (flag & REF_ISSYMREF)\n+\t\treturn error(\"refname %s is a symbolic ref, renaming it is not supported\",\n+\t\t\toldref);\n+\tif (!symref)\n+\t\treturn error(\"refname %s not found\", oldref);\n+\n+\tif (!rename_ref_available(oldref, newref))\n+\t\treturn 1;\n+\n+\tif (log) {\n+\t\tstruct strbuf old_log_sentinel = STRBUF_INIT;\n+\t\tMDB_val key;\n+\t\tint log_all;\n+\n+\t\tlog_all = log_all_ref_updates;\n+\t\tlog_all_ref_updates = 1;\n+\t\tif (safe_create_reflog(newref, 0, &err)) {\n+\t\t\terror(\"can't create reflog for %s: %s\", newref, err.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tlog_all_ref_updates = log_all;\n+\n+\t\tfor_each_reflog_ent(oldref, rename_reflog_ent, (void *)newref);\n+\t\tstrbuf_addf(&old_log_sentinel, \"logs/%sxxxxxxxx\", oldref);\n+\t\tmemset(old_log_sentinel.buf + old_log_sentinel.len - 8, 0, 8);\n+\n+\t\tkey.mv_size = old_log_sentinel.len;\n+\t\tkey.mv_data = old_log_sentinel.buf;\n+\t\t/* It's OK if the old reflog is missing */\n+\t\tmdb_del_or_die(&transaction.info, &key, NULL);\n+\n+\t\tstrbuf_release(&old_log_sentinel);\n+\t}\n+\n+\tif (ref_transaction_delete(ref_transaction, oldref,\n+\t\torig_sha1, REF_NODEREF, NULL, &err)) {\n+\t\terror(\"unable to delete old %s\", oldref);\n+\t\treturn 1;\n+\t}\n+\n+\tif (lmdb_transaction_update(ref_transaction, newref, orig_sha1, NULL,\n+\t\t\t\t 0, logmsg, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn 1;\n+\t}\n+\n+\tif (lmdb_transaction_commit(ref_transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+\n+}\n+\n+static int lmdb_for_each_reflog_ent_order(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data, int reverse)\n+{\n+\tMDB_val key, val;\n+\tchar *search_key;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum MDB_cursor_op direction = reverse ? MDB_PREV : MDB_NEXT;\n+\tuint64_t zero = 0ULL;\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsearch_key = xmalloc(len + 1);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\tstrcpy(search_key, log_path);\n+\n+\tif (reverse) {\n+\t\t/*\n+\t\t * For a reverse search, start at the key\n+\t\t * lexicographically after the searched-for key.\n+\t\t * That's the one with \\001 appended to the key.\n+\t\t */\n+\n+\t\tsearch_key[len - 1] = 1;\n+\t\tsearch_key[len] = 0;\n+\t\tkey.mv_size = len + 1;\n+\t} else {\n+\t\tkey.mv_size = len;\n+\t}\n+\n+\tkey.mv_data = search_key;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\n+\tmdb_cursor_open_or_die(&transaction.info, &cursor);\n+\n+\ttransaction.cursor = cursor;\n+\n+\t/*\n+\t * MDB's cursor API requires that the first mdb_cursor_get be\n+\t * called with MDB_SET_RANGE. For reverse searches, this will\n+\t * give us the entry one-past the entry we're looking for, so\n+\t * we should jump back using MDB_PREV.\n+\t */\n+\tmdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\tif (direction == MDB_PREV)\n+\t\tmdb_cursor_get_or_die(cursor, &key, &val, direction);\n+\n+\tdo {\n+\t\tif (key.mv_size < len)\n+\t\t\tbreak;\n+\n+\t\tif (!starts_with(key.mv_data, log_path) || ((char *)key.mv_data)[len - 1] != 0)\n+\t\t\tbreak;\n+\n+\t\tif (!memcmp(&zero, ((char *)key.mv_data) + key.mv_size - 8, 8))\n+\t\t\tcontinue;\n+\n+\t\tassert(val.mv_size != 0);\n+\n+\t\tstrbuf_add(&sb, val.mv_data, val.mv_size - 1);\n+\t\tret = show_one_reflog_ent(&sb, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\n+\t\tstrbuf_reset(&sb);\n+\t} while (!mdb_cursor_get_or_die(cursor, &key, &val, direction));\n+\n+\tstrbuf_release(&sb);\n+\tfree(log_path);\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_reflog_ent(const char *refname,\n+\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn files_for_each_reflog_ent(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 0);\n+}\n+\n+static int lmdb_for_each_reflog_ent_reverse(const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn files_for_each_reflog_ent_reverse(refname, fn, cb_data);\n+\treturn lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 1);\n+}\n+\n+static int lmdb_reflog_exists(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint ret = 1;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn files_reflog_exists(refname);\n+\n+\tlen = strlen(refname) + 6;\n+\tlog_path = xmalloc(len);\n+\tsprintf(log_path, \"logs/%s\", refname);\n+\n+\tkey.mv_data = log_path;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tmdb_cursor_open_or_die(&transaction.info, &cursor);\n+\n+\tif (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE)) {\n+\t\tret = 0;\n+\t} else if (!starts_with(key.mv_data, log_path)) {\n+\t\tret = 0;\n+\t}\n+\n+\tfree(log_path);\n+\tmdb_cursor_close(cursor);\n+\n+\treturn ret;\n+}\n+\n+struct wrapped_each_ref_fn {\n+\teach_ref_fn *fn;\n+\tvoid *cb_data;\n+};\n+\n+static int check_reflog(const char *refname,\n+\t\t\tconst struct object_id *oid, int flags, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn *wrapped = cb_data;\n+\n+\tif (reflog_exists(refname))\n+\t\treturn wrapped->fn(refname, oid, 0, wrapped->cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_for_each_reflog(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct wrapped_each_ref_fn wrapped = {fn, cb_data};\n+\tint result = head_ref(fn, cb_data);\n+\tif (result)\n+\t return result;\n+\treturn for_each_ref(check_reflog, &wrapped);\n+}\n+\n+static int lmdb_create_reflog(const char *refname, int force_create, struct strbuf *err)\n+{\n+\t/*\n+\t * We mark that there is a reflog by creating a key of the\n+\t * form logs/$refname followed by nine \\0 (one for\n+\t * string-termination, 8 in lieu of a timestamp), with an empty\n+\t * value.\n+\t */\n+\n+\tint in_transaction = in_write_transaction();\n+\tMDB_val key, val;\n+\n+\tif (!force_create && !should_autocreate_reflog(refname))\n+\t\treturn 0;\n+\n+\tif (!in_transaction)\n+\t\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tkey.mv_size = strlen(refname) + 5 + 1 + 8;\n+\tkey.mv_data = xcalloc(1, key.mv_size);\n+\tsprintf((char *)key.mv_data, \"logs/%s\", refname);\n+\tval.mv_size = 0;\n+\tval.mv_data = NULL;\n+\tmdb_put_or_die(&transaction.info, &key, &val, 0);\n+\n+\tfree(key.mv_data);\n+\tif (!in_transaction)\n+\t\treturn lmdb_transaction_commit(\n+\t\t\t(struct ref_transaction *)&transaction, err);\n+\treturn 0;\n+}\n+\n+struct expire_reflog_cb {\n+\tunsigned int flags;\n+\treflog_expiry_should_prune_fn *should_prune_fn;\n+\tvoid *policy_cb;\n+\tunsigned char last_kept_sha1[20];\n+};\n+\n+static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t const char *email, unsigned long timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\tstruct expire_reflog_cb *cb = cb_data;\n+\tstruct expire_reflog_policy_cb *policy_cb = cb->policy_cb;\n+\n+\tif (cb->flags & EXPIRE_REFLOGS_REWRITE)\n+\t\tosha1 = cb->last_kept_sha1;\n+\n+\tif ((*cb->should_prune_fn)(osha1, nsha1, email, timestamp, tz,\n+\t\t\t\t message, policy_cb)) {\n+\t\tif (cb->flags & EXPIRE_REFLOGS_DRY_RUN)\n+\t\t\tprintf(\"would prune %s\", message);\n+\t\telse {\n+\t\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\t\tprintf(\"prune %s\", message);\n+\n+\t\t\tmdb_cursor_del_or_die(transaction.cursor, 0);\n+\t\t}\n+\t} else {\n+\t\thashcpy(cb->last_kept_sha1, nsha1);\n+\t\tif (cb->flags & EXPIRE_REFLOGS_VERBOSE)\n+\t\t\tprintf(\"keep %s\", message);\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_ref(const char *refname, const unsigned char *sha1)\n+{\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_transaction *transaction;\n+\n+\ttransaction = lmdb_transaction_begin(&err);\n+\tif (!transaction) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (lmdb_transaction_update(transaction, refname, sha1, NULL,\n+\t\t\t\t REF_NO_REFLOG, NULL, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\tif (lmdb_transaction_commit(transaction, &err)) {\n+\t\terror(\"%s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int lmdb_reflog_expire(const char *refname, const unsigned char *sha1,\n+\t\t unsigned int flags,\n+\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t void *policy_cb_data)\n+{\n+\tstruct expire_reflog_cb cb;\n+\tint dry_run = flags & EXPIRE_REFLOGS_DRY_RUN;\n+\tint status = 0;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char resolved_sha1[20];\n+\tint type;\n+\n+\tif (ref_type(refname) != REF_TYPE_NORMAL)\n+\t\treturn files_reflog_expire(refname, sha1, flags, prepare_fn,\n+\t\t\t\t\t should_prune_fn, cleanup_fn,\n+\t\t\t\t\t policy_cb_data);\n+\n+\tmemset(&cb, 0, sizeof(cb));\n+\tcb.flags = flags;\n+\tcb.policy_cb = policy_cb_data;\n+\tcb.should_prune_fn = should_prune_fn;\n+\n+\tlmdb_transaction_begin_flags_or_die(dry_run ? MDB_RDONLY : 0);\n+\n+\tcheck_ref(transaction.info.txn, refname, sha1,\n+\t\t resolved_sha1, 0, &type);\n+\n+\t(*prepare_fn)(refname, sha1, cb.policy_cb);\n+\tlmdb_for_each_reflog_ent(refname, expire_reflog_ent, &cb);\n+\t(*cleanup_fn)(cb.policy_cb);\n+\n+\tif (!dry_run) {\n+\t\t/*\n+\t\t * It doesn't make sense to adjust a reference pointed\n+\t\t * to by a symbolic ref based on expiring entries in\n+\t\t * the symbolic reference's reflog. Nor can we update\n+\t\t * a reference if there are no remaining reflog\n+\t\t * entries.\n+\t\t */\n+\t\tint update = (flags & EXPIRE_REFLOGS_UPDATE_REF) &&\n+\t\t\t!(type & REF_ISSYMREF) &&\n+\t\t\t!is_null_sha1(cb.last_kept_sha1);\n+\n+\t\tif (lmdb_transaction_commit(\n+\t\t\t (struct ref_transaction *)&transaction, &err)) {\n+\t\t\tstatus |= error(\"couldn't write logs/%s: %s\", refname,\n+\t\t\t\t\terr.buf);\n+\t\t\tstrbuf_release(&err);\n+\t\t} else if (update &&\n+\t\t\t write_ref(refname, cb.last_kept_sha1)) {\n+\t\t\tstatus |= error(\"couldn't set %s\",\n+\t\t\t\t\trefname);\n+\t\t}\n+\t}\n+\treturn status;\n+}\n+\n+static int lmdb_pack_refs(unsigned int flags)\n+{\n+\t/* This concept does not exist in this backend. */\n+\treturn 0;\n+}\n+\n+static int lmdb_peel_ref(const char *refname, unsigned char *sha1)\n+{\n+\tint flag;\n+\tunsigned char base[20];\n+\n+\tif (read_ref_full(refname, RESOLVE_REF_READING, base, &flag))\n+\t\treturn -1;\n+\n+\treturn peel_object(base, sha1);\n+}\n+\n+static int lmdb_create_symref(struct ref_transaction *trans,\n+\t\t\t const char *ref_target,\n+\t\t\t const char *refs_heads_master,\n+\t\t\t const char *logmsg)\n+{\n+\n+\tstruct strbuf err = STRBUF_INIT;\n+\tunsigned char old_sha1[20], new_sha1[20];\n+\tMDB_val key, val;\n+\tchar *valdata;\n+\tstruct lmdb_transaction *transaction = (struct lmdb_transaction *)trans;\n+\n+\tif (logmsg && read_ref(ref_target, old_sha1))\n+\t\thashclr(old_sha1);\n+\n+\tkey.mv_size = strlen(ref_target) + 1;\n+\tkey.mv_data = xstrdup(ref_target);\n+\n+\tval.mv_size = strlen(refs_heads_master) + 1 + 5;\n+\tvaldata = xmalloc(val.mv_size);\n+\tsprintf(valdata, \"ref: %s\", refs_heads_master);\n+\tval.mv_data = valdata;\n+\n+\tmdb_put_or_die(&transaction->info, &key, &val, 0);\n+\n+\tif (logmsg && !read_ref(refs_heads_master, new_sha1) &&\n+\t log_ref_write(ref_target, old_sha1, new_sha1, logmsg, 0, &err)) {\n+\t\terror(\"log_ref_write failed: %s\", err.buf);\n+\t\tstrbuf_release(&err);\n+\t}\n+\n+\tfree(key.mv_data);\n+\tfree(valdata);\n+\n+\treturn 0;\n+}\n+\n+MDB_env *submodule_txn_begin(struct lmdb_transaction_info *info)\n+{\n+\tint ret;\n+\tMDB_env *submodule_env = NULL;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_git_path_submodule(&path, info->submodule, \"refdb\");\n+\n+\tif (!is_directory(path.buf))\n+\t\tgoto done;\n+\n+\tmkdir(path.buf, 0775);\n+\n+\tinit_env(&submodule_env, path.buf);\n+\n+\tif ((ret = mdb_txn_begin(submodule_env, NULL, MDB_RDONLY, &info->txn)) != MDB_SUCCESS) {\n+\t\tdie(\"mdb_txn_begin failed: %s\", mdb_strerror(ret));\n+\n+\t}\n+\tif ((ret = mdb_dbi_open(info->txn, NULL, 0, &info->dbi)) != MDB_SUCCESS) {\n+\t\tdie(\"mdb_txn_open failed: %s\", mdb_strerror(ret));\n+\t}\n+\n+done:\n+\tstrbuf_release(&path);\n+\treturn submodule_env;\n+}\n+\n+static int lmdb_resolve_gitlink_ref(const char *submodule, const char *refname,\n+\t\t\t\t unsigned char *sha1)\n+{\n+\tstruct lmdb_transaction_info info;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tinfo.txn = NULL;\n+\tinfo.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&info);\n+\tif (!submodule_env)\n+\t\treturn -1;\n+\tresult = !resolve_ref_unsafe_txn(&info, refname,\n+\t\t\t\t\t RESOLVE_REF_READING, sha1, NULL);\n+\n+\tmdb_txn_abort(info.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result ? -1 : 0;\n+}\n+\n+static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (submodule) {\n+\t\tif (resolve_gitlink_ref(submodule, \"HEAD\", oid.hash) == 0)\n+\t\t\treturn fn(\"HEAD\", &oid, 0, cb_data);\n+\n+\t\treturn 0;\n+\t}\n+\n+\tif (!read_ref_full(\"HEAD\", RESOLVE_REF_READING, oid.hash, &flag))\n+\t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n+\n+\treturn 0;\n+}\n+\n+static int lmdb_head_ref(each_ref_fn fn, void *cb_data)\n+{\n+\treturn do_head_ref(NULL, fn, cb_data);\n+}\n+\n+static int lmdb_head_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_head_ref(submodule, fn, cb_data);\n+}\n+\n+/*\n+ * Call fn for each reference for which the refname begins with base.\n+ * If trim is non-zero, then trim that many characters off the\n+ * beginning of each refname before passing the refname to fn. flags\n+ * can be DO_FOR_EACH_INCLUDE_BROKEN to include broken references in\n+ * the iteration. If fn ever returns a non-zero value, stop the\n+ * iteration and return that value; otherwise, return 0.\n+ */\n+static int do_for_each_ref(struct lmdb_transaction_info *info,\n+\t\t\t const char *base, each_ref_fn fn, int trim,\n+\t\t\t int flags, void *cb_data)\n+{\n+\n+\tMDB_val key, val;\n+\tMDB_cursor *cursor;\n+\tint baselen;\n+\tchar *search_key;\n+\tint retval;\n+\tint mdb_ret;\n+\n+\tretval = do_for_each_per_worktree_ref(info->submodule, base, fn, trim,\n+\t\t\t\t\t flags, cb_data);\n+\tif (retval)\n+\t\treturn retval;\n+\n+\tif (ref_paranoia < 0)\n+\t\tref_paranoia = git_env_bool(\"GIT_REF_PARANOIA\", 0);\n+\tif (ref_paranoia)\n+\t\tflags |= DO_FOR_EACH_INCLUDE_BROKEN;\n+\n+\tif (!base || !*base) {\n+\t\tbase = \"refs/\";\n+\t\ttrim = 0;\n+\t}\n+\n+\tbaselen = strlen(base);\n+\tsearch_key = xmalloc(baselen + 1);\n+\tstrcpy(search_key, base);\n+\tkey.mv_size = baselen + 1;\n+\tkey.mv_data = search_key;\n+\n+\tmdb_cursor_open_or_die(info, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\n+\twhile (!mdb_ret) {\n+\t\tstruct object_id oid;\n+\t\tint parsed_flags = 0;\n+\n+\t\tif (memcmp(key.mv_data, base, baselen))\n+\t\t\tbreak;\n+\n+\t\tparse_ref_data(info, key.mv_data + (trim ? baselen : 0),\n+\t\t\t val.mv_data, oid.hash, 0, &parsed_flags, 0);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_BROKEN ||\n+\t\t (!(parsed_flags & REF_ISBROKEN) &&\n+\t\t has_sha1_file(oid.hash))) {\n+\t\t\tretval = fn(key.mv_data + (trim ? baselen : 0), &oid, parsed_flags, cb_data);\n+\t\t\tif (retval)\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tmdb_cursor_close(cursor);\n+\tfree(search_key);\n+\n+\treturn retval;\n+}\n+\n+static int lmdb_for_each_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, \"\", fn, 0, 0, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_submodule(const char *submodule, each_ref_fn fn,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tstruct lmdb_transaction_info info;\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref(fn, cb_data);\n+\n+\tinfo.txn = NULL;\n+\tinfo.submodule = submodule;\n+\n+\tsubmodule_env = submodule_txn_begin(&info);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&info, \"\", fn, 0, 0, cb_data);\n+\tmdb_txn_abort(info.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_ref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, prefix, fn, strlen(prefix),\n+\t\t\t 0, cb_data);\n+}\n+\n+static int lmdb_for_each_fullref_in(const char *prefix, each_ref_fn fn,\n+\t\t\t\t void *cb_data, unsigned int broken)\n+{\n+\tunsigned int flag = 0;\n+\n+\tif (broken)\n+\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, prefix, fn, 0, flag, cb_data);\n+}\n+\n+static int lmdb_for_each_ref_in_submodule(const char *submodule,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t each_ref_fn fn, void *cb_data)\n+{\n+\tstruct lmdb_transaction_info info = {NULL};\n+\tMDB_env *submodule_env;\n+\tint result;\n+\n+\tif (!submodule)\n+\t\treturn for_each_ref_in(prefix, fn, cb_data);\n+\n+\tinfo.submodule = submodule;\n+\tsubmodule_env = submodule_txn_begin(&info);\n+\tif (!submodule_env)\n+\t\treturn 0;\n+\tresult = do_for_each_ref(&info, prefix, fn,\n+\t\t\t\t strlen(prefix), 0, cb_data);\n+\tmdb_txn_abort(info.txn);\n+\tmdb_env_close(submodule_env);\n+\treturn result;\n+}\n+\n+static int lmdb_for_each_replace_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, git_replace_ref_base, fn,\n+\t\t\t strlen(git_replace_ref_base), 0, cb_data);\n+}\n+\n+static int lmdb_for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken, void *cb_data)\n+{\n+\tunsigned int flag = 0;\n+\n+\tif (broken)\n+\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, type, fn, 0, flag, cb_data);\n+}\n+\n+static int lmdb_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret;\n+\tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tret = do_for_each_ref(&transaction.info, buf.buf, fn, 0, 0, cb_data);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n+static int lmdb_for_each_rawref(each_ref_fn fn, void *cb_data)\n+{\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\treturn do_for_each_ref(&transaction.info, \"\", fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_read(const char *key)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup;\n+\tint ret;\n+\n+\tlmdb_transaction_begin_flags_or_die(MDB_RDONLY);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_get_or_die(&transaction.info, &key_val, &val);\n+\tfree(keydup);\n+\tswitch (ret) {\n+\tcase 0:\n+\t\tprintf(\"%s\\n\", (char *)val.mv_data);\n+\t\treturn 0;\n+\tcase MDB_NOTFOUND:\n+\t\tfprintf(stderr, \"%s not found\\n\", key);\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 2;\n+\t}\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_write(const char *key, const char *value)\n+{\n+\tMDB_val key_val, val;\n+\tchar *keydup, *valdup;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tvaldup = xstrdup(value);\n+\tval.mv_data = valdup;\n+\tval.mv_size = strlen(value) + 1;\n+\n+\tmdb_put_or_die(&transaction.info, &key_val, &val, 0);\n+\tassert(lmdb_transaction_commit((struct ref_transaction *)&transaction,\n+\t\t\t\t &err) == 0);\n+\n+\tfree(keydup);\n+\tfree(valdup);\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_delete(const char *key)\n+{\n+\tMDB_val key_val;\n+\tchar *keydup;\n+\tint ret;\n+\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\tkeydup = xstrdup(key);\n+\tkey_val.mv_data = keydup;\n+\tkey_val.mv_size = strlen(key) + 1;\n+\n+\tret = mdb_del_or_die(&transaction.info, &key_val, NULL);\n+\n+\tassert(lmdb_transaction_commit((struct lmdb_transaction *)&transaction,\n+\t\t\t\t &err) == 0);\n+\n+\tfree(keydup);\n+\treturn ret;\n+}\n+\n+static int print_raw_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n+\t\t\t\tconst char *email, unsigned long timestamp,\n+\t\t\t\tint tz, const char *message, void *cb_data)\n+{\n+\tint *any = cb_data;\n+\t*any = 1;\n+\n+\tif (*message != '\\n')\n+\t\tprintf(\"%s %s %s %lu %+05d\\t%s\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz, message);\n+\telse\n+\t\tprintf(\"%s %s %s %lu %+05d\\n\", sha1_to_hex(osha1),\n+\t\t sha1_to_hex(nsha1),\n+\t\t email, timestamp, tz);\n+\treturn 0;\n+}\n+\n+/* For testing only! */\n+int test_refdb_raw_reflog(const char *refname)\n+{\n+\tMDB_val key, val;\n+\tchar *search_key;\n+\tchar *log_path;\n+\tint len;\n+\tMDB_cursor *cursor;\n+\tint result = 1;\n+\tint any = 0;\n+\n+\tfor_each_reflog_ent(refname, print_raw_reflog_ent, &any);\n+\n+\treturn !any;\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_delete_reflog(char *refname)\n+{\n+\tMDB_val key, val;\n+\tint mdb_ret;\n+\tchar *search_key;\n+\tMDB_cursor *cursor;\n+\tint len;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tif (refname) {\n+\t\tlen = strlen(refname) + 5 + 1; /* logs/ + 0*/\n+\t\tsearch_key = xmalloc(len);\n+\t\tsprintf(search_key, \"logs/%s\", refname);\n+\t} else {\n+\t\tlen = 6; /* logs/ + 0*/\n+\t\tsearch_key = xstrdup(\"logs/\");\n+\t}\n+\tkey.mv_data = search_key;\n+\tkey.mv_size = len;\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\tmdb_cursor_open_or_die(&transaction.info, &cursor);\n+\n+\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);\n+\twhile (!mdb_ret) {\n+\t\tif (!starts_with(key.mv_data, search_key))\n+\t\t\tbreak;\n+\t\tif (refname && ((char *)val.mv_data)[len - 1] == 0)\n+\t\t\tbreak;\n+\n+\t\tmdb_cursor_del_or_die(cursor, 0);\n+\t\tmdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);\n+\t}\n+\n+\tfree(search_key);\n+\tmdb_cursor_close(cursor);\n+\n+\tassert(lmdb_transaction_commit(\n+\t\t (struct ref_transaction *)&transaction, &err) == 0);\n+\treturn;\n+}\n+\n+static void format_lmdb_reflog_ent(struct strbuf *dst, struct strbuf *src)\n+{\n+\tunsigned char osha1[20], nsha1[20];\n+\tconst char *msg;\n+\n+\tget_sha1_hex(src->buf, osha1);\n+\tget_sha1_hex(src->buf + 41, nsha1);\n+\n+\tmsg = strchr(src->buf + 82, '\\n');\n+\tassert(msg);\n+\tmsg += 1;\n+\n+\tformat_reflog_entry(dst, osha1, nsha1, src->buf + 82, msg);\n+}\n+\n+/* For testing only! */\n+void test_refdb_raw_append_reflog(const char *refname)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tuint64_t now = getnanotime();\n+\tMDB_val key, val;\n+\tstruct strbuf err = STRBUF_INIT;\n+\n+\tkey.mv_size = strlen(refname) + 14;\n+\tkey.mv_data = xcalloc(1, key.mv_size);\n+\tsprintf(key.mv_data, \"logs/%s\", refname);\n+\n+\tlmdb_transaction_begin_flags_or_die(0);\n+\n+\t/* We do not remove the header here, because this is just for\n+\t * tests, so it's OK to be a bit inefficient */\n+\n+\twhile (strbuf_getwholeline(&input, stdin, '\\n') != EOF) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t/* \"logs/\" + \\0 + 8-byte timestamp for sorting and expiry */\n+\t\twrite_u64(key.mv_data + key.mv_size - 8, htonll(now++));\n+\n+\t\t/* Convert the input from files-reflog format to\n+\t\t * lmdb-reflog-format */\n+\n+\t\tformat_lmdb_reflog_ent(&sb, &input);\n+\t\tval.mv_data = sb.buf;\n+\t\tval.mv_size = sb.len + 1;\n+\t\tmdb_put_or_die(&transaction.info, &key, &val, 0);\n+\t\tstrbuf_reset(&sb);\n+\t\tinput.len = 0;\n+\t}\n+\n+\tassert(lmdb_transaction_commit(&transaction, &err) == 0);\n+\tfree(key.mv_data);\n+}\n+\n+struct ref_be refs_be_lmdb = {\n+\tNULL,\n+\t\"lmdb\",\n+\tlmdb_init_backend,\n+\tlmdb_initdb,\n+\tlmdb_transaction_begin,\n+\tlmdb_transaction_update,\n+\tlmdb_transaction_create,\n+\tlmdb_transaction_delete,\n+\tlmdb_transaction_verify,\n+\tlmdb_transaction_commit,\n+\tlmdb_transaction_commit, /* initial commit */\n+\tlmdb_transaction_free,\n+\tlmdb_rename_ref,\n+\tlmdb_for_each_reflog_ent,\n+\tlmdb_for_each_reflog_ent_reverse,\n+\tlmdb_for_each_reflog,\n+\tlmdb_reflog_exists,\n+\tlmdb_create_reflog,\n+\tlmdb_delete_reflog,\n+\tlmdb_reflog_expire,\n+\tlmdb_resolve_ref_unsafe,\n+\tlmdb_verify_refname_available,\n+\tlmdb_pack_refs,\n+\tlmdb_peel_ref,\n+\tlmdb_create_symref,\n+\tlmdb_resolve_gitlink_ref,\n+\tlmdb_head_ref,\n+\tlmdb_head_ref_submodule,\n+\tlmdb_for_each_ref,\n+\tlmdb_for_each_ref_submodule,\n+\tlmdb_for_each_ref_in,\n+\tlmdb_for_each_fullref_in,\n+\tlmdb_for_each_ref_in_submodule,\n+\tlmdb_for_each_rawref,\n+\tlmdb_for_each_namespaced_ref,\n+\tlmdb_for_each_replace_ref,\n+\tlmdb_for_each_reftype_fullpath,\n+};\ndiff --git a/refs.h b/refs.h\nindex ac88371..f499093 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -641,6 +641,15 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n \t\t\t void *policy_cb_data);\n \n+/*\n+ * Read the refdb configuration data out of the config file\n+ */\n+struct refdb_config_data {\n+\tconst char *refs_backend_type;\n+\tconst char *refs_base;\n+};\n+int refdb_config(const char *var, const char *value, void *ptr);\n+\n /* refs backends */\n typedef void (*ref_backend_init_fn)(void *data);\n typedef int (*ref_backend_initdb_fn)(struct strbuf *err, int shared);\n@@ -763,6 +772,7 @@ struct ref_be {\n \n \n extern struct ref_be refs_be_files;\n+extern struct ref_be refs_be_lmdb;\n int set_refs_backend(const char *name, void *init_data);\n void register_refs_backend(struct ref_be *be);\n \ndiff --git a/setup.c b/setup.c\nindex b264471..3e31112 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"refs.h\"\n #include \"string-list.h\"\n \n static int inside_git_dir = -1;\n@@ -262,16 +263,26 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n \treturn ret;\n }\n \n+int refdb_config(const char *var, const char *value, void *ptr)\n+{\n+ struct refdb_config_data *cdata = ptr;\n+\n+ if (!strcmp(var, \"core.refs-backend-type\"))\n+\t cdata->refs_backend_type = strdup((char *)value);\n+ return 0;\n+}\n+\n /*\n * Test if it looks like we're at a git directory.\n * We want to see:\n *\n * - either an objects/ directory _or_ the proper\n * GIT_OBJECT_DIRECTORY environment variable\n- * - a refs/ directory\n- * - either a HEAD symlink or a HEAD file that is formatted as\n- * a proper \"ref:\", or a regular file HEAD that has a properly\n- * formatted sha1 object name.\n+ * - a refdb/ directory or\n+ * - a refs/ directory\n+ * - either a HEAD symlink or a HEAD file that is formatted as\n+ * a proper \"ref:\", or a regular file HEAD that has a properly\n+ * formatted sha1 object name.\n */\n int is_git_directory(const char *suspect)\n {\n@@ -302,8 +313,13 @@ int is_git_directory(const char *suspect)\n \n \tstrbuf_setlen(&path, len);\n \tstrbuf_addstr(&path, \"/refs\");\n-\tif (access(path.buf, X_OK))\n-\t\tgoto done;\n+\n+\tif (access(path.buf, X_OK)) {\n+\t\tstrbuf_setlen(&path, len);\n+\t\tstrbuf_addstr(&path, \"/refdb\");\n+\t\tif (access(path.buf, X_OK))\n+\t\t\tgoto done;\n+\t}\n \n \tret = 1;\n done:\ndiff --git a/test-refs-be-lmdb.c b/test-refs-be-lmdb.c\nnew file mode 100644\nindex 0000000..bb875ff\n--- /dev/null\n+++ b/test-refs-be-lmdb.c\n@@ -0,0 +1,68 @@\n+#include \"cache.h\"\n+#include \"string-list.h\"\n+#include \"parse-options.h\"\n+#include \"refs.h\"\n+\n+static const char * const test_refs_be_lmdb_usage[] = {\n+\t\"git test-refs-be-lmdb \",\n+\t\"git test-refs-be-lmdb \",\n+\tNULL,\n+};\n+\n+int test_refdb_raw_read(const char *key);\n+void test_refdb_raw_write(const char *key, const char *value);\n+int test_refdb_raw_reflog(const char *refname);\n+int test_refdb_raw_delete(const char *key);\n+void test_refdb_raw_delete_reflog(const char *refname);\n+void test_refdb_raw_append_reflog(const char *refname);\n+\n+int main(int argc, const char **argv)\n+{\n+\tconst char *delete = NULL;\n+\tconst char *reflog = NULL;\n+\tconst char *append_reflog = NULL;\n+\tint delete_missing_error = 0;\n+\tint clear_reflog = 0;\n+\tstruct refdb_config_data config_data = {NULL};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING('d', NULL, &delete, \"branch\", \"delete refdb entry\"),\n+\t\tOPT_STRING('l', NULL, &reflog, \"branch\", \"show reflog\"),\n+\t\tOPT_STRING('a', NULL, &append_reflog, \"branch\", \"append to reflog\"),\n+\t\tOPT_BOOL('c', NULL, &clear_reflog, \"delete reflog. If a branch is provided, the reflog for that branch will be deleted; else all reflogs will be deleted.\"),\n+\t\tOPT_BOOL('x', NULL, &delete_missing_error,\n+\t\t\t \"deleting a missing key is an error\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, \"\", options, test_refs_be_lmdb_usage,\n+\t\t\t 0);\n+\n+\tif (!append_reflog && !clear_reflog && !delete && !reflog && argc != 1 && argc != 2)\n+\t\tusage_with_options(test_refs_be_lmdb_usage,\n+\t\t\t\t options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tconfig_data.refs_backend_type = \"lmdb\";\n+\tconfig_data.refs_base = get_git_dir();\n+\n+\tregister_refs_backend(&refs_be_lmdb);\n+\tset_refs_backend(\"lmdb\", &config_data);\n+\n+\tif (clear_reflog) {\n+\t\ttest_refdb_raw_delete_reflog(argv[0]);\n+\t} else if (append_reflog) {\n+\t\ttest_refdb_raw_append_reflog(append_reflog);\n+\t} else if (reflog) {\n+\t\treturn test_refdb_raw_reflog(reflog);\n+\t} else if (delete) {\n+\t\tif (test_refdb_raw_delete(delete) && delete_missing_error)\n+\t\t\treturn 1;\n+\t} else if (argc == 1) {\n+\t\treturn test_refdb_raw_read(argv[0]);\n+\t} else {\n+\t\ttest_refdb_raw_write(argv[0], argv[1]);\n+\t}\n+\treturn 0;\n+}\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"270861","messageId":"1443477738-32023-44-git-send-email-dturner@twopensource.com","threadId":"40445","inReplyTo":"1443477738-32023-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v2 43/43] refs: tests for db backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-09-28T22:02:18Z","receivedAt":"2015-09-28T22:02:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add tests for the database backend.\n\nSigned-off-by: David Turner \n---\n t/t1460-refs-be-db.sh | 1103 ++++++++++++++++++++++++++++++++++++++++++\n t/t1470-refs-be-db-reflog.sh | 353 ++++++++++++++\n 2 files changed, 1456 insertions(+)\n create mode 100755 t/t1460-refs-be-db.sh\n create mode 100755 t/t1470-refs-be-db-reflog.sh\n\ndiff --git a/t/t1460-refs-be-db.sh b/t/t1460-refs-be-db.sh\nnew file mode 100755\nindex 0000000..f13b0f0\n--- /dev/null\n+++ b/t/t1460-refs-be-db.sh\n@@ -0,0 +1,1103 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2015 Twitter, Inc\n+# Copyright (c) 2006 Shawn Pearce\n+# This test is based on t1400-update-ref.sh\n+#\n+\n+test_description='Test lmdb refs backend'\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+raw_ref() {\n+\ttest-refs-be-lmdb \"$1\"\n+}\n+\n+delete_ref() {\n+\ttest-refs-be-lmdb -d \"$1\"\n+}\n+\n+write_ref() {\n+\ttest-refs-be-lmdb \"$1\" \"$2\"\n+}\n+\n+raw_reflog() {\n+\ttest-refs-be-lmdb -l \"$1\"\n+}\n+\n+delete_all_reflogs() {\n+\ttest-refs-be-lmdb -c\n+}\n+\n+append_reflog() {\n+\ttest-refs-be-lmdb -a \"$1\"\n+}\n+\n+Z=$_z40\n+\n+test_expect_success setup '\n+\tgit init --refs-backend-type=lmdb &&\n+\tfor name in A B C D E F\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tsha1=$(echo $name | git commit-tree $T) &&\n+\t\teval $name=$sha1\n+\tdone\n+'\n+\n+m=refs/heads/master\n+n_dir=refs/heads/gu\n+n=$n_dir/fixes\n+\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m\" \\\n+\t\"git update-ref $m $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m with stale ref\" '\n+\ttest_must_fail git update-ref -d $m $A &&\n+\ttest $B = \"$(raw_ref $m)\"\n+'\n+test_expect_success \"delete $m\" '\n+\tgit update-ref -d $m $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \"delete $m without oldvalue verification\" \"\n+\tgit update-ref $m $A &&\n+\ttest $A = \\$(raw_ref $m) &&\n+\tgit update-ref -d $m &&\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"fail to create $n\" \\\n+\t\"git update-ref $n_dir $A &&\n+\t test_must_fail git update-ref $n $A >out 2>err\"\n+\n+delete_ref $n_dir\n+rm -f out err\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"fail to delete $m (by HEAD) with stale ref\" '\n+\ttest_must_fail git update-ref -d HEAD $A &&\n+\ttest $B = $(raw_ref '\"$m\"')\n+'\n+test_expect_success \"delete $m (by HEAD)\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"pack refs\" \\\n+\t\"git pack-refs --all\"\n+test_expect_success \\\n+\t\"move $m (by HEAD)\" \\\n+\t\"git update-ref HEAD $B $A &&\n+\t test $B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \"delete $m (by HEAD) should remove both packed and loose $m\" '\n+\tgit update-ref -d HEAD $B &&\n+\t! raw_ref $m\n+'\n+delete_ref $m\n+\n+OLD_HEAD=$(raw_ref HEAD)\n+test_expect_success \"delete symref without dereference\" '\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+\n+test_expect_success \"delete symref without dereference when the referred ref is packed\" '\n+\techo foo >foo.c &&\n+\tgit add foo.c &&\n+\tgit commit -m foo &&\n+\tgit pack-refs --all &&\n+\tgit update-ref --no-deref -d HEAD &&\n+\t! raw_ref HEAD\n+'\n+write_ref HEAD \"$OLD_HEAD\"\n+delete_ref $m\n+\n+test_expect_success 'update-ref -d is not confused by self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\ttest_must_fail git update-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete self-reference' '\n+\tgit symbolic-ref refs/heads/self refs/heads/self &&\n+\ttest_when_finished \"delete_ref refs/heads/self\" &&\n+\tgit update-ref --no-deref -d refs/heads/self\n+'\n+\n+test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '\n+\ttest-refs-be-lmdb refs/heads/bad \"\" &&\n+\ttest_when_finished \"delete_ref refs/heads/bad\" &&\n+\tgit symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&\n+\ttest_when_finished \"delete_ref refs/heads/ref-to-bad\" &&\n+\traw_ref refs/heads/ref-to-bad &&\n+\tgit update-ref --no-deref -d refs/heads/ref-to-bad &&\n+\t! raw_ref refs/heads/ref-to-bad\n+'\n+\n+test_expect_success '(not) create HEAD with old sha1' \"\n+\ttest_must_fail git update-ref HEAD $A $B\n+\"\n+test_expect_success \"(not) prior created .git/$m\" \"\n+\t! raw_ref $m\n+\"\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create HEAD\" \\\n+\t\"git update-ref HEAD $A\"\n+test_expect_success '(not) change HEAD with wrong SHA1' \"\n+\ttest_must_fail git update-ref HEAD $B $Z\n+\"\n+test_expect_success \"(not) changed .git/$m\" \"\n+\t! test $B\"' = $(raw_ref '\"$m\"')\n+'\n+\n+: a repository with working tree always has reflog these days...\n+delete_all_reflogs\n+: | append_reflog $m\n+delete_ref $m\n+\n+test_expect_success \\\n+\t\"create $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:30\" \\\n+\t git update-ref HEAD '\"$A\"' -m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:31\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by touch)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:41\" \\\n+\t git update-ref HEAD'\" $A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150200 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t\"raw_reflog $m >actual &&\n+\t test_cmp expect actual\"\n+delete_ref $m\n+delete_all_reflogs\n+: | append_reflog $m\n+rm -f actual expect\n+\n+test_expect_success \\\n+\t'enable core.logAllRefUpdates' \\\n+\t'git config core.logAllRefUpdates true &&\n+\t test true = $(git config --bool --get core.logAllRefUpdates)'\n+\n+test_expect_success \\\n+\t\"create $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:32\" \\\n+\t git update-ref HEAD'\" $A \"'-m \"Initial Creation\" &&\n+\t test '\"$A\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"update $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:33\" \\\n+\t git update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t test '\"$B\"' = $(raw_ref '\"$m\"')'\n+test_expect_success \\\n+\t\"set $m (logged by config)\" \\\n+\t'GIT_COMMITTER_DATE=\"2005-05-26 23:43\" \\\n+\t git update-ref HEAD '\"$A &&\n+\t test $A\"' = $(raw_ref '\"$m\"')'\n+\n+cat >expect < 1117150320 +0000\tInitial Creation\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 +0000\tSwitch\n+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 +0000\n+EOF\n+test_expect_success \\\n+\t\"verifying $m's log\" \\\n+\t'raw_reflog $m >actual &&\n+\ttest_cmp expect actual'\n+delete_ref $m\n+rm -f expect\n+\n+git update-ref $m $D\n+git reflog expire --expire=all $m\n+\n+append_reflog $m < 1117150320 -0500\n+$C $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150350 -0500\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 -0500\n+$F $Z $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150680 -0500\n+$Z $E $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 -0500\n+EOF\n+\n+ed=\"Thu, 26 May 2005 18:32:00 -0500\"\n+gd=\"Thu, 26 May 2005 18:33:00 -0500\"\n+ld=\"Thu, 26 May 2005 18:43:00 -0500\"\n+test_expect_success \\\n+\t'Query \"master@{May 25 2005}\" (before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 25 2005}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t\"Query master@{2005-05-25} (before history)\" \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify master@{2005-05-25} >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t echo test \"warning: Log for '\\'master\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:31:59}\" (1 second before history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:31:59}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"warning: Log for '\\''master'\\'' only goes back to $ed.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:00}\" (exactly history start)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:00}\" >o 2>e &&\n+\t test '\"$C\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{May 26 2005 23:32:30}\" (first non-creation change)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{May 26 2005 23:32:30}\" >o 2>e &&\n+\t test '\"$A\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:33:01}\" (middle of history with gap)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:33:01}\" >o 2>e &&\n+\t test '\"$B\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m has gap after $gd\"'.\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:38:00}\" (middle of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:38:00}\" >o 2>e &&\n+\t test '\"$Z\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-26 23:43:00}\" (exact end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-26 23:43:00}\" >o 2>e &&\n+\t test '\"$E\"' = $(cat o) &&\n+\t test \"\" = \"$(cat e)\"'\n+test_expect_success \\\n+\t'Query \"master@{2005-05-28}\" (past end of history)' \\\n+\t'rm -f o e &&\n+\t git rev-parse --verify \"master@{2005-05-28}\" >o 2>e &&\n+\t test '\"$D\"' = $(cat o) &&\n+\t test \"warning: Log for ref '\"$m unexpectedly ended on $ld\"'.\" = \"$(cat e)\"'\n+\n+\n+git reflog expire --expire=all $m\n+delete_ref $m\n+\n+test_expect_success \\\n+ 'creating initial files' \\\n+ 'echo TEST >F &&\n+ git add F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:30\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:30\" git commit -m add -a &&\n+\t h_TEST=$(git rev-parse --verify HEAD) &&\n+\t echo The other day this did not work. >M &&\n+\t echo And then Bob told me how to fix it. >>M &&\n+\t echo OTHER >F &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:41\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:41\" git commit -F M -a &&\n+\t h_OTHER=$(git rev-parse --verify HEAD) &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:44\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:44\" git commit --amend &&\n+\t h_FIXED=$(git rev-parse --verify HEAD) &&\n+\t echo Merged initial commit and a later commit. >M &&\n+\t echo $h_TEST >.git/MERGE_HEAD &&\n+\t GIT_AUTHOR_DATE=\"2005-05-26 23:45\" \\\n+\t GIT_COMMITTER_DATE=\"2005-05-26 23:45\" git commit -F M &&\n+\t h_MERGED=$(git rev-parse --verify HEAD) &&\n+\t rm -f M'\n+\n+cat >expect < 1117150200 +0000\tcommit (initial): add\n+$h_TEST $h_OTHER $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\tcommit: The other day this did not work.\n+$h_OTHER $h_FIXED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151040 +0000\tcommit (amend): The other day this did not work.\n+$h_FIXED $h_MERGED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151100 +0000\tcommit (merge): Merged initial commit and a later commit.\n+EOF\n+test_expect_success \\\n+\t'git commit logged updates' \\\n+\t\"raw_reflog $m >actual &&\n+\ttest_cmp expect actual\"\n+unset h_TEST h_OTHER h_FIXED h_MERGED\n+\n+test_expect_success \\\n+\t'git cat-file blob master:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob master:F)'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:30}:F (expect TEST)' \\\n+\t'test TEST = $(git cat-file blob \"master@{2005-05-26 23:30}:F\")'\n+test_expect_success \\\n+\t'git cat-file blob master@{2005-05-26 23:42}:F (expect OTHER)' \\\n+\t'test OTHER = $(git cat-file blob \"master@{2005-05-26 23:42}:F\")'\n+\n+a=refs/heads/a\n+b=refs/heads/b\n+c=refs/heads/c\n+E='\"\"'\n+F='%s\\0'\n+pws='path with space'\n+\n+test_expect_success 'stdin test setup' '\n+\techo \"$pws\" >\"$pws\" &&\n+\tgit add -- \"$pws\" &&\n+\tgit commit -m \"$pws\"\n+'\n+\n+test_expect_success '-z fails without --stdin' '\n+\ttest_must_fail git update-ref -z $m $m $m 2>err &&\n+\tgrep \"usage: git update-ref\" err\n+'\n+\n+test_expect_success 'stdin works with no input' '\n+\t>stdin &&\n+\tgit update-ref --stdin stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin fails on only whitespace' '\n+\techo \" \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin fails on leading whitespace' '\n+\techo \" create $a $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a $m\" err\n+'\n+\n+test_expect_success 'stdin fails on unknown command' '\n+\techo \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin fails on unbalanced quotes' '\n+\techo \"create $a \\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails on invalid escape' '\n+\techo \"create $a \\\"ma\\zter\\\"\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: badly quoted argument: \\\\\\\"ma\\\\\\\\zter\\\\\\\"\" err\n+'\n+\n+test_expect_success 'stdin fails on junk after quoted argument' '\n+\techo \"create \\\"$a\\\"master\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: unexpected character after quoted argument: \\\\\\\"$a\\\\\\\"master\" err\n+'\n+\n+test_expect_success 'stdin fails create with no ref' '\n+\techo \"create \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with no new value' '\n+\techo \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails create with too many arguments' '\n+\techo \"create $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails update with no ref' '\n+\techo \"update \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with no new value' '\n+\techo \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: missing \" err\n+'\n+\n+test_expect_success 'stdin fails update with too many arguments' '\n+\techo \"update $a $m $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails delete with no ref' '\n+\techo \"delete \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin fails delete with too many arguments' '\n+\techo \"delete $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails verify with too many arguments' '\n+\techo \"verify $a $m $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: verify $a: extra input: $m\" err\n+'\n+\n+test_expect_success 'stdin fails option with unknown name' '\n+\techo \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin fails with duplicate refs' '\n+\tcat >stdin <<-EOF &&\n+\tcreate $a $m\n+\tcreate $b $m\n+\tcreate $a $m\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin create ref works' '\n+\techo \"create $a $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with quoted argument' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"$m\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin succeeds with escaped character' '\n+\tgit update-ref -d $a &&\n+\techo \"create $a \\\"ma\\\\163ter\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update ref creates with zero old value' '\n+\techo \"update $b $m $Z\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin update ref creates with empty old value' '\n+\techo \"update $b $m $E\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin create ref works with path with space to blob' '\n+\techo \"create refs/blobs/pws \\\"$m:$pws\\\"\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin update ref fails with wrong old value' '\n+\techo \"update $c $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref fails with bad old value' '\n+\techo \"update $c $m does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with bad new value' '\n+\techo \"create $c does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin create ref fails with zero new value' '\n+\techo \"create $c \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: create $c: zero \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update ref works with right old value' '\n+\techo \"update $b $m~1 $m\" >stdin &&\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with wrong old value' '\n+\techo \"delete $a $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref fails with zero old value' '\n+\techo \"delete $a \" >stdin &&\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tupdate refs/TESTSYMREF $a $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tcat >stdin <<-EOF &&\n+\toption no-deref\n+\tdelete refs/TESTSYMREF $b\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete ref works with right old value' '\n+\techo \"delete $b $m~1\" >stdin &&\n+\tgit update-ref --stdin stdin <<-EOF &&\n+\tupdate $a $m\n+\tcreate $b $m\n+\tverify $c\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $m\" >stdin &&\n+\tgit update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify succeeds for missing reference' '\n+\techo \"verify refs/heads/missing $Z\" >stdin &&\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref --stdin expect &&\n+\techo \"verify $m $m~1\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m $Z\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\techo \"verify $m\" >stdin &&\n+\ttest_must_fail git update-ref --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin update refs works with identity updates' '\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c $Z $E\n+\tEOF\n+\tgit update-ref --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tcat >stdin <<-EOF &&\n+\tupdate $a $m $m\n+\tupdate $b $m $m\n+\tupdate $c ''\n+\tEOF\n+\ttest_must_fail git update-ref --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tcat >stdin <<-EOF &&\n+\tdelete $a $m\n+\tupdate $b $Z $m\n+\tupdate $c $E $m~1\n+\tEOF\n+\tgit update-ref --stdin stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on empty command' '\n+\tprintf $F \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: empty command in input\" err\n+'\n+\n+test_expect_success 'stdin -z fails on only whitespace' '\n+\tprintf $F \" \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: \" err\n+'\n+\n+test_expect_success 'stdin -z fails on leading whitespace' '\n+\tprintf $F \" create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: whitespace before command: create $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails on unknown command' '\n+\tprintf $F \"unknown $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: unknown $a\" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no ref' '\n+\tprintf $F \"create \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with no new value' '\n+\tprintf $F \"create $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails create with too many arguments' '\n+\tprintf $F \"create $a\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no ref' '\n+\tprintf $F \"update \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too few args' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z emits warning with empty new value' '\n+\tgit update-ref $a $m &&\n+\tprintf $F \"update $a\" \"\" \"\" >stdin &&\n+\tgit update-ref -z --stdin err &&\n+\tgrep \"warning: update $a: missing , treating as zero\" err &&\n+\ttest_must_fail git rev-parse --verify -q $a\n+'\n+\n+test_expect_success 'stdin -z fails update with no new value' '\n+\tprintf $F \"update $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with no old value' '\n+\tprintf $F \"update $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails update with too many arguments' '\n+\tprintf $F \"update $m\" \"$m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no ref' '\n+\tprintf $F \"delete \" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete: missing \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with no old value' '\n+\tprintf $F \"delete $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails delete with too many arguments' '\n+\tprintf $F \"delete $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with too many arguments' '\n+\tprintf $F \"verify $m\" \"$m\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: unknown command: $m\" err\n+'\n+\n+test_expect_success 'stdin -z fails verify with no old value' '\n+\tprintf $F \"verify $a\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: verify $a: unexpected end of input when reading \" err\n+'\n+\n+test_expect_success 'stdin -z fails option with unknown name' '\n+\tprintf $F \"option unknown\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: option unknown: unknown\" err\n+'\n+\n+test_expect_success 'stdin -z fails with duplicate refs' '\n+\tprintf $F \"create $a\" \"$m\" \"create $b\" \"$m\" \"create $a\" \"$m\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: Multiple updates for ref '\"'\"'$a'\"'\"' not allowed.\" err\n+'\n+\n+test_expect_success 'stdin -z create ref works' '\n+\tprintf $F \"create $a\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update ref creates with zero old value' '\n+\tprintf $F \"update $b\" \"$m\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d $b\n+'\n+\n+test_expect_success 'stdin -z update ref creates with empty old value' '\n+\tprintf $F \"update $b\" \"$m\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref works with path with space to blob' '\n+\tprintf $F \"create refs/blobs/pws\" \"$m:$pws\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse refs/blobs/pws >actual &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/blobs/pws\n+'\n+\n+test_expect_success 'stdin -z update ref fails with wrong old value' '\n+\tprintf $F \"update $c\" \"$m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref fails with bad old value' '\n+\tprintf $F \"update $c\" \"$m\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: update $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails when ref exists' '\n+\tgit update-ref $c $m &&\n+\tgit rev-parse \"$c\" >expect &&\n+\tprintf $F \"create $c\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse \"$c\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z create ref fails with bad new value' '\n+\tgit update-ref -d \"$c\" &&\n+\tprintf $F \"create $c\" \"does-not-exist\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: invalid : does-not-exist\" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z create ref fails with empty new value' '\n+\tprintf $F \"create $c\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: create $c: missing \" err &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update ref works with right old value' '\n+\tprintf $F \"update $b\" \"$m~1\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with wrong old value' '\n+\tprintf $F \"delete $a\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$a'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref fails with zero old value' '\n+\tprintf $F \"delete $a\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: delete $a: zero \" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"update refs/TESTSYMREF\" \"$a\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $m~1 >expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete symref works option no-deref' '\n+\tgit symbolic-ref refs/TESTSYMREF $b &&\n+\tprintf $F \"option no-deref\" \"delete refs/TESTSYMREF\" \"$b\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete ref works with right old value' '\n+\tprintf $F \"delete $b\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for correct value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$m\" >stdin &&\n+\tgit update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify succeeds for missing reference' '\n+\tprintf $F \"verify refs/heads/missing\" \"$Z\" >stdin &&\n+\tgit update-ref -z --stdin stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tprintf $F \"verify $m\" \"$m~1\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken null value' '\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z verify fails for mistaken empty value' '\n+\tM=$(git rev-parse $m) &&\n+\ttest_when_finished \"git update-ref $m $M\" &&\n+\tgit rev-parse $m >expect &&\n+\tprintf $F \"verify $m\" \"\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z update refs works with identity updates' '\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$Z\" \"\" >stdin &&\n+\tgit update-ref -z --stdin expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_fail git rev-parse --verify -q $c\n+'\n+\n+test_expect_success 'stdin -z update refs fails with wrong old value' '\n+\tgit update-ref $c $m &&\n+\tprintf $F \"update $a\" \"$m\" \"$m\" \"update $b\" \"$m\" \"$m\" \"update $c\" \"$m\" \"$Z\" >stdin &&\n+\ttest_must_fail git update-ref -z --stdin err &&\n+\tgrep \"fatal: cannot lock the ref '\"'\"'$c'\"'\"'\" err &&\n+\tgit rev-parse $m >expect &&\n+\tgit rev-parse $a >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $b >actual &&\n+\ttest_cmp expect actual &&\n+\tgit rev-parse $c >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stdin -z delete refs works with packed and loose refs' '\n+\tgit pack-refs --all &&\n+\tgit update-ref $c $m~1 &&\n+\tprintf $F \"delete $a\" \"$m\" \"update $b\" \"$Z\" \"$m\" \"update $c\" \"\" \"$m~1\" >stdin &&\n+\tgit update-ref -z --stdin /dev/null || test-refs-be-lmdb -l \"$1\"\n+}\n+\n+append_reflog() {\n+\ttest-refs-be-lmdb -a \"$1\"\n+}\n+\n+check_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\" && git cat-file -t $o || {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone &&\n+\ttest -z \"$gaah\"\n+}\n+\n+check_fsck () {\n+\toutput=$(git fsck --full)\n+\tcase \"$1\" in\n+\t'')\n+\t\ttest -z \"$output\" ;;\n+\t*)\n+\t\techo \"$output\" | grep \"$1\" ;;\n+\tesac\n+}\n+\n+corrupt () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmv .git/objects/$aa/$zz .git/$aa$zz\n+}\n+\n+recover () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmkdir -p .git/objects/$aa\n+\tmv .git/$aa$zz .git/objects/$aa/$zz\n+}\n+\n+check_dont_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\"\n+\t\tgit cat-file -t $o && {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone\n+\ttest -z \"$gaah\"\n+}\n+\n+test_expect_success setup '\n+\tgit init --refs-backend-type=lmdb &&\n+\tmkdir -p A/B &&\n+\techo rat >C &&\n+\techo ox >A/D &&\n+\techo tiger >A/B/E &&\n+\tgit add . &&\n+\n+\ttest_tick && git commit -m rabbit &&\n+\tH=`git rev-parse --verify HEAD` &&\n+\tA=`git rev-parse --verify HEAD:A` &&\n+\tB=`git rev-parse --verify HEAD:A/B` &&\n+\tC=`git rev-parse --verify HEAD:C` &&\n+\tD=`git rev-parse --verify HEAD:A/D` &&\n+\tE=`git rev-parse --verify HEAD:A/B/E` &&\n+\tcheck_fsck &&\n+\n+\ttest_chmod +x C &&\n+\tgit add C &&\n+\ttest_tick && git commit -m dragon &&\n+\tL=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f C A/B/E &&\n+\techo snake >F &&\n+\techo horse >A/G &&\n+\tgit add F A/G &&\n+\ttest_tick && git commit -a -m sheep &&\n+\tF=`git rev-parse --verify HEAD:F` &&\n+\tG=`git rev-parse --verify HEAD:A/G` &&\n+\tI=`git rev-parse --verify HEAD:A` &&\n+\tJ=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f A/G &&\n+\ttest_tick && git commit -a -m monkey &&\n+\tK=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tcheck_fsck &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success rewind '\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\ttest -f C &&\n+\ttest -f A/B/E &&\n+\t! test -f F &&\n+\t! test -f A/G &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K L &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog\n+'\n+\n+test_expect_success 'corrupt and check' '\n+\n+\tcorrupt $F &&\n+\tcheck_fsck \"missing blob $F\"\n+\n+'\n+\n+test_expect_success 'reflog expire --dry-run should not touch reflog' '\n+\n+\tgit reflog expire --dry-run \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 5 reflog &&\n+\n+\tcheck_fsck \"missing blob $F\"\n+'\n+\n+test_expect_success 'reflog expire' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\techo git reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 2 reflog &&\n+\n+\tcheck_fsck \"dangling commit $K\"\n+'\n+\n+test_expect_success 'prune and fsck' '\n+\n+\tgit prune &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E H L &&\n+\tcheck_dont_have F G I J K\n+\n+'\n+\n+test_expect_success 'recover and check' '\n+\n+\trecover $F &&\n+\tcheck_fsck \"dangling blob $F\"\n+\n+'\n+\n+test_expect_success 'delete' '\n+\techo 1 > C &&\n+\ttest_tick &&\n+\tgit commit -m rat C &&\n+\n+\techo 2 > C &&\n+\ttest_tick &&\n+\tgit commit -m ox C &&\n+\n+\techo 3 > C &&\n+\ttest_tick &&\n+\tgit commit -m tiger C &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\tmaster_entry_count=$(git reflog show master | wc -l) &&\n+\n+\ttest $HEAD_entry_count = 5 &&\n+\ttest $master_entry_count = 5 &&\n+\n+\n+\tgit reflog delete master@{1} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\ttest $HEAD_entry_count = $(git reflog | wc -l) &&\n+\t! grep ox < output &&\n+\n+\tmaster_entry_count=$(wc -l < output) &&\n+\n+\tgit reflog delete HEAD@{1} &&\n+\ttest $(($HEAD_entry_count -1)) = $(git reflog | wc -l) &&\n+\ttest $master_entry_count = $(git reflog show master | wc -l) &&\n+\n+\tHEAD_entry_count=$(git reflog | wc -l) &&\n+\n+\tgit reflog delete master@{07.04.2005.15:15:00.-0700} &&\n+\tgit reflog show master > output &&\n+\ttest $(($master_entry_count - 1)) = $(wc -l < output) &&\n+\t! grep dragon < output\n+\n+'\n+\n+test_expect_success 'rewind2' '\n+\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success '--expire=never' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=never \\\n+\t\t--expire-unreachable=never \\\n+\t\t--all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=never' '\n+\n+\tgit config gc.reflogexpire never &&\n+\tgit config gc.reflogexpireunreachable never &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog\n+'\n+\n+test_expect_success 'gc.reflogexpire=false' '\n+\n+\tgit config gc.reflogexpire false &&\n+\tgit config gc.reflogexpireunreachable false &&\n+\tgit reflog expire --verbose --all &&\n+\traw_reflog refs/heads/master >reflog &&\n+\ttest_when_finished rm -f reflog &&\n+\ttest_line_count = 4 reflog &&\n+\n+\tgit config --unset gc.reflogexpire &&\n+\tgit config --unset gc.reflogexpireunreachable\n+\n+'\n+\n+test_expect_success 'checkout should not delete log for packed ref' '\n+\ttest $(git reflog master | wc -l) = 4 &&\n+\tgit branch foo &&\n+\tgit pack-refs --all &&\n+\tgit checkout foo &&\n+\ttest $(git reflog master | wc -l) = 4\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs on)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# now logs/refs/heads/one is a stale directory, but\n+\t# we should move it out of the way to create \"one\" reflog\n+\tgit branch one master &&\n+\techo \"one@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs off)' '\n+\ttest_when_finished \"git branch -d one || git branch -d one/two\" &&\n+\n+\tgit branch one/two master &&\n+\techo \"one/two@{0} branch: Created from master\" >expect &&\n+\tgit log -g --format=\"%gd %gs\" one/two >actual &&\n+\ttest_cmp expect actual &&\n+\tgit branch -d one/two &&\n+\n+\t# same as before, but we only create a reflog for \"one\" if\n+\t# it already exists, which it does not\n+\tgit -c core.logallrefupdates=false branch one master &&\n+\t: >expect &&\n+\tgit log -g --format=\"%gd %gs\" one >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Triggering the bug detected by this test requires a newline to fall\n+# exactly BUFSIZ-1 bytes from the end of the file. We don't know\n+# what that value is, since it's platform dependent. However, if\n+# we choose some value N, we also catch any D which divides N evenly\n+# (since we will read backwards in chunks of D). So we choose 8K,\n+# which catches glibc (with an 8K BUFSIZ) and *BSD (1K).\n+#\n+# Each line is 114 characters, so we need 75 to still have a few before the\n+# last 8K. The 89-character padding on the final entry lines up our\n+# newline exactly.\n+test_expect_success 'parsing reverse reflogs at BUFSIZ boundaries' '\n+\tgit checkout -b reflogskip &&\n+\tz38=00000000000000000000000000000000000000 &&\n+\tident=\"abc 0000000001 +0000\" &&\n+\tfor i in $(test_seq 1 75); do\n+\t\tprintf \"$z38%02d $z38%02d %s\\t\" $i $(($i+1)) \"$ident\" &&\n+\t\tif test $i = 75; then\n+\t\t\tfor j in $(test_seq 1 89); do\n+\t\t\t\tprintf X\n+\t\t\tdone\n+\t\telse\n+\t\t\tprintf X\n+\t\tfi &&\n+\t\tprintf \"\\n\"\n+\tdone | append_reflog refs/heads/reflogskip &&\n+\tgit rev-parse reflogskip@{73} >actual &&\n+\techo ${z38}03 >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.4.2.644.g97b850b-twtrsrc\n"},{"id":"271035","messageId":"xmqqh9m93s5y.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"1443477738-32023-28-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 27/43] refs.c: move should_autocreate_reflog to common code","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-02T20:57:13Z","receivedAt":"2015-10-02T20:57:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Up to this step everything looked sensible.\n\nThanks.\n"},{"id":"271038","messageId":"xmqq1tdd3qe9.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"1443477738-32023-43-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-02T21:35:26Z","receivedAt":"2015-10-02T21:35:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> diff --git a/refs-be-lmdb.c b/refs-be-lmdb.c\n> new file mode 100644\n> index 0000000..99cbd29\n> --- /dev/null\n> +++ b/refs-be-lmdb.c\n> @@ -0,0 +1,2003 @@\n> +/*\n> + ...\n> + */\n> +#include \n> +#include \n> +#include \"cache.h\"\n\n\"git-compat-util.h\" (or \"cache.h\", because it is well known and\nincludes \"git-compat-util.h\" as the first thing before doing\nanything else) must be the first file included in any of our C\nfiles, unless it is a platform specific compat/ object.\n"},{"id":"271041","messageId":"560F6172.3040404@web.de","threadId":"40445","inReplyTo":"1443477738-32023-2-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-10-03T05:02:42Z","receivedAt":"2015-10-03T05:02:42Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 29.09.15 00:01, David Turner wrote:\n>\n(Not sure if this is the right thread to report on)\n\nIn file included from builtin/commit.c:20:\n./refs.h:695:16: warning: redefinition of typedef 'ref_transaction_free_fn' is a C11 feature\n [-Wtypedef-redefinition]\ntypedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n ^\n"},{"id":"271060","messageId":"1443891000.7753.4.camel@twopensource.com","threadId":"40445","inReplyTo":"560F6172.3040404@web.de","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-03T16:50:00Z","receivedAt":"2015-10-03T16:50:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2015-10-03 at 07:02 +0200, Torsten Bögershausen wrote:\n> On 29.09.15 00:01, David Turner wrote:\n> >\n> (Not sure if this is the right thread to report on)\n> \n> In file included from builtin/commit.c:20:\n> ./refs.h:695:16: warning: redefinition of typedef 'ref_transaction_free_fn' is a C11 feature\n> [-Wtypedef-redefinition]\n> typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n> ^\n\n\nFixed, thanks.\n\nWhat compiler flag did you turn on to see that warning?\n"},{"id":"271066","messageId":"20151003173930.GA25904@spirit","threadId":"40445","inReplyTo":"1443477738-32023-44-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 43/43] refs: tests for db backend","fromName":"Dennis Kaarsemaker","fromEmail":"dennis@kaarsemaker.net","sentAt":"2015-10-03T17:39:31Z","receivedAt":"2015-10-03T17:39:31Z","isPatch":true,"sender":{"key":"dennis@kaarsemaker.net","avatar":"https://avatars.githubusercontent.com/u/200649?v=4"},"body":"On Mon, Sep 28, 2015 at 06:02:18PM -0400, David Turner wrote:\n> Add tests for the database backend.\n> \n> Signed-off-by: David Turner \n> ---\n> t/t1460-refs-be-db.sh | 1103 ++++++++++++++++++++++++++++++++++++++++++\n> t/t1470-refs-be-db-reflog.sh | 353 ++++++++++++++\n> 2 files changed, 1456 insertions(+)\n> create mode 100755 t/t1460-refs-be-db.sh\n> create mode 100755 t/t1470-refs-be-db-reflog.sh\n\nThese break 'make test' on builds without the db backend. Maybe squash\nin something like the following:\n\ndiff --git a/t/t1460-refs-be-db.sh b/t/t1460-refs-be-db.sh\nindex f13b0f0..c8222ed 100755\n--- a/t/t1460-refs-be-db.sh\n+++ b/t/t1460-refs-be-db.sh\n@@ -9,6 +9,11 @@ test_description='Test lmdb refs backend'\n TEST_NO_CREATE_REPO=1\n . ./test-lib.sh\n \n+if ! test -e ../../test-refs-be-lmdb; then\n+\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n+\ttest_done\n+fi\n+\n raw_ref() {\n \ttest-refs-be-lmdb \"$1\"\n }\ndiff --git a/t/t1470-refs-be-db-reflog.sh b/t/t1470-refs-be-db-reflog.sh\nindex 99a705d..2538a58 100755\n--- a/t/t1470-refs-be-db-reflog.sh\n+++ b/t/t1470-refs-be-db-reflog.sh\n@@ -8,6 +8,11 @@ test_description='Test prune and reflog expiration'\n TEST_NO_CREATE_REPO=1\n . ./test-lib.sh\n \n+if ! test -e ../../test-refs-be-lmdb; then\n+\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n+\ttest_done\n+fi\n+\n raw_reflog() {\n \tcat .git/logs/$1 2>/dev/null || test-refs-be-lmdb -l \"$1\"\n }\n\n\nAlso, test 18 in t1460 is broken:\nexpecting success: \n git symbolic-ref refs/heads/self refs/heads/self &&\n test_when_finished \"delete_ref refs/heads/self\" &&\n test_must_fail git update-ref -d refs/heads/self\n\ntest_must_fail: command succeeded: git update-ref -d refs/heads/self\nnot ok 18 - update-ref -d is not confused by self-reference\n# \n# git symbolic-ref refs/heads/self refs/heads/self &&\n# test_when_finished \"delete_ref refs/heads/self\" &&\n# test_must_fail git update-ref -d refs/heads/self\n# \n\n-- \nDennis Kaarsemaker \nhttp://twitter.com/seveas\n"},{"id":"271071","messageId":"56104375.3030904@web.de","threadId":"40445","inReplyTo":"1443891000.7753.4.camel@twopensource.com","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-10-03T21:07:01Z","receivedAt":"2015-10-03T21:07:01Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 03.10.15 18:50, David Turner wrote:\n> On Sat, 2015-10-03 at 07:02 +0200, Torsten Bögershausen wrote:\n>> On 29.09.15 00:01, David Turner wrote:\n>>>\n>> (Not sure if this is the right thread to report on)\n>>\n>> In file included from builtin/commit.c:20:\n>> ./refs.h:695:16: warning: redefinition of typedef 'ref_transaction_free_fn' is a C11 feature\n>> [-Wtypedef-redefinition]\n>> typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n>> ^\n> \n> \n> Fixed, thanks.\n> \n> What compiler flag did you turn on to see that warning?\nMac OS X, 10.9:\n\n gcc --version\nConfigured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1\nApple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)\nTarget: x86_64-apple-darwin13.4.0\nThread model: posix\n"},{"id":"271079","messageId":"5610C206.4090307@web.de","threadId":"40445","inReplyTo":"1443891000.7753.4.camel@twopensource.com","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-10-04T06:07:02Z","receivedAt":"2015-10-04T06:07:02Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 2015-10-03 18.50, David Turner wrote:\n> On Sat, 2015-10-03 at 07:02 +0200, Torsten Bögershausen wrote:\n>> On 29.09.15 00:01, David Turner wrote:\n>> (Not sure if this is the right thread to report on)\n>>\n>> In file included from builtin/commit.c:20:\n>> ./refs.h:695:16: warning: redefinition of typedef 'ref_transaction_free_fn' is a C11 feature\n>> [-Wtypedef-redefinition]\n>> typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n>> ^\n>\n> Fixed, thanks.\n>\n> What compiler flag did you turn on to see that warning?\nThis compiler reported it as an error, not a warning:\ngcc --version\ni686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)\nCopyright (C) 2007 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions. There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n"},{"id":"271118","messageId":"5611FCA4.9030809@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-2-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T04:29:24Z","receivedAt":"2015-10-05T04:29:24Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:01 AM, David Turner wrote:\n> From: Ronnie Sahlberg \n> \n> Create a public version of verify_refname_available that backends can\n> provide.\n> \n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> refs.c | 51 +++++++++++++++++++++++++++++----------------------\n> refs.h | 17 +++++++++++++++++\n> 2 files changed, 46 insertions(+), 22 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 132eff5..ce551e9 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -279,7 +279,7 @@ struct ref_dir {\n> * presence of an empty subdirectory does not block the creation of a\n> * similarly-named reference. (The fact that reference names with the\n> * same leading components can conflict *with each other* is a\n> - * separate issue that is regulated by verify_refname_available().)\n> + * separate issue that is regulated by verify_refname_available_dir().)\n> *\n> * Please note that the name field contains the fully-qualified\n> * reference (or subdirectory) name. Space could be saved by only\n> @@ -911,11 +911,11 @@ static int nonmatching_ref_fn(struct ref_entry *entry, void *vdata)\n> *\n> * extras and skip must be sorted.\n> */\n> -static int verify_refname_available(const char *refname,\n> -\t\t\t\t const struct string_list *extras,\n> -\t\t\t\t const struct string_list *skip,\n> -\t\t\t\t struct ref_dir *dir,\n> -\t\t\t\t struct strbuf *err)\n> +static int verify_refname_available_dir(const char *refname,\n> +\t\t\t\t\tconst struct string_list *extras,\n> +\t\t\t\t\tconst struct string_list *skip,\n> +\t\t\t\t\tstruct ref_dir *dir,\n> +\t\t\t\t\tstruct strbuf *err)\n> {\n> \tconst char *slash;\n> \tint pos;\n> [...]\n> diff --git a/refs.h b/refs.h\n> index 6d30c98..79ea220 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -218,6 +218,23 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st\n> int pack_refs(unsigned int flags);\n> \n> /*\n> + * Return true iff a reference named refname could be created without\n> + * conflicting with the name of an existing reference. If\n> + * skip is non-NULL, ignore potential conflicts with refs in skip\n> + * (e.g., because they are scheduled for deletion in the same\n> + * operation).\n> + *\n> + * Two reference names conflict if one of them exactly matches the\n> + * leading components of the other; e.g., \"foo/bar\" conflicts with\n> + * both \"foo\" and with \"foo/bar/baz\" but not with \"foo/bar\" or\n> + * \"foo/barbados\".\n> + *\n> + * skip must be sorted.\n> + */\n\nThis comment is approximately a copy of the comment for\nverify_refname_available_dir(). It seems unnecessary to keep both of\nthem (and is also a small maintenance burden). I suggest you shorten the\ncomment at verify_refname_available_dir() and make it refer to the\ncomment for this function for the details.\n\n> +int verify_refname_available(const char *newname, struct string_list *extra,\n> +\t\t\t struct string_list *skip, struct strbuf *err);\n> +\n> +/*\n> * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.\n> * REF_NODEREF: act on the ref directly, instead of dereferencing\n> * symbolic references.\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271119","messageId":"5611FDD9.1050405@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-3-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 02/43] refs: make repack_without_refs and is_branch public","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T04:34:33Z","receivedAt":"2015-10-05T04:34:33Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:01 AM, David Turner wrote:\n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> refs.c | 9 +--------\n> refs.h | 13 +++++++++++++\n> 2 files changed, 14 insertions(+), 8 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index ce551e9..2741cc5 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2826,14 +2826,7 @@ int pack_refs(unsigned int flags)\n> \treturn 0;\n> }\n> \n> -/*\n> - * Rewrite the packed-refs file, omitting any refs listed in\n> - * 'refnames'. On error, leave packed-refs unchanged, write an error\n> - * message to 'err', and return a nonzero value.\n> - *\n> - * The refs in 'refnames' needn't be sorted. `err` must not be NULL.\n> - */\n> -static int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n> +int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n\nI looked for the corresponding change to remove `static` from\nis_branch(). Apparently that function already had external linkage, even\nthough it was not listed in the header file. As a convenience to readers\nyou might note that peculiarity in the commit message.\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271127","messageId":"56122EB6.2020501@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-19-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 18/43] refs-be-files.c: add a backend method structure with transaction functions","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T08:03:02Z","receivedAt":"2015-10-05T08:03:02Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:01 AM, David Turner wrote:\n> From: Ronnie Sahlberg \n> \n> Add a ref structure for backend methods. Start by adding method pointers\n> for the transaction functions.\n> \n> Add a function set_refs_backend to switch between backends. The files\n> based backend is the default.\n> \n> Signed-off-by: Ronnie Sahlberg \n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 62 +++++++++++++++++++++++++++-------------------\n> refs.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> refs.h | 36 +++++++++++++++++++++++++++\n> 3 files changed, 150 insertions(+), 25 deletions(-)\n> \n> [...]\n> diff --git a/refs.h b/refs.h\n> index 729bc3c..a1db3ef 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -530,4 +530,40 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n> \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n> \t\t\t void *policy_cb_data);\n> \n> +/* refs backends */\n> +typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n\nHmmm, I thought our convention was to define typedefs for functions\nthemselves, not for the pointer-to-function; e.g.,\n\n typedef struct ref_transaction *ref_transaction_begin_fn(struct\nstrbuf *err);\n\n(which would require `struct ref_be` to be changed to\n\n ref_transaction_begin_fn *transaction_begin;\n\netc.) But now as I grep through the code it looks like both conventions\nare used. So never mind :-)\n\n> +typedef int (*ref_transaction_update_fn)(struct ref_transaction *transaction,\n> +\t\tconst char *refname, const unsigned char *new_sha1,\n> +\t\tconst unsigned char *old_sha1, unsigned int flags,\n> +\t\tconst char *msg, struct strbuf *err);\n> +typedef int (*ref_transaction_create_fn)(\n> +\t\tstruct ref_transaction *transaction,\n> +\t\tconst char *refname, const unsigned char *new_sha1,\n> +\t\tunsigned int flags, const char *msg, struct strbuf *err);\n> +typedef int (*ref_transaction_delete_fn)(struct ref_transaction *transaction,\n> +\t\tconst char *refname, const unsigned char *old_sha1,\n> +\t\tunsigned int flags, const char *msg, struct strbuf *err);\n> +typedef int (*ref_transaction_verify_fn)(struct ref_transaction *transaction,\n> +\t\tconst char *refname, const unsigned char *old_sha1,\n> +\t\tunsigned int flags, struct strbuf *err);\n> +typedef int (*ref_transaction_commit_fn)(struct ref_transaction *transaction,\n> +\t\t\t\t struct strbuf *err);\n> +typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n> +\n> +struct ref_be {\n> +\tstruct ref_be *next;\n> +\tconst char *name;\n> +\tref_transaction_begin_fn transaction_begin;\n> +\tref_transaction_update_fn transaction_update;\n> +\tref_transaction_create_fn transaction_create;\n> +\tref_transaction_delete_fn transaction_delete;\n> +\tref_transaction_verify_fn transaction_verify;\n> +\tref_transaction_commit_fn transaction_commit;\n> +\tref_transaction_free_fn transaction_free;\n> +};\n> +\n> +\n> +extern struct ref_be refs_be_files;\n> +int set_refs_backend(const char *name);\n> +\n> #endif /* REFS_H */\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271128","messageId":"56123279.4060605@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-23-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 22/43] refs-be-files.c: add do_for_each_per_worktree_ref","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T08:19:05Z","receivedAt":"2015-10-05T08:19:05Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:01 AM, David Turner wrote:\n> Alternate refs backends might still use files to store per-worktree\n> refs. So the files backend's ref-loading infrastructure should be\n> available to those backends, just for use on per-worktree refs. Add\n> do_for_each_per_worktree_ref, which iterates over per-worktree refs.\n> \n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 15 ++++++++++++---\n> refs.h | 11 +++++++++++\n> 2 files changed, 23 insertions(+), 3 deletions(-)\n> \n> diff --git a/refs-be-files.c b/refs-be-files.c\n> index eb18a20..dc89289 100644\n> --- a/refs-be-files.c\n> +++ b/refs-be-files.c\n> @@ -587,9 +587,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n> \tdir->sorted = dir->nr = i;\n> }\n> \n> -/* Include broken references in a do_for_each_ref*() iteration: */\n> -#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n> -\n> /*\n> * Return true iff the reference described by entry can be resolved to\n> * an object in the database. Emit a warning if the referred-to\n> [...]\n> diff --git a/refs.h b/refs.h\n> index 5875fe5..09d140d 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -152,6 +152,12 @@ struct ref_transaction;\n> */\n> #define REF_BAD_NAME 0x08\n> \n> +/* Include broken references in a do_for_each_ref*() iteration */\n> +#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n\nWhy do you move this definition from refs-be-files.c?\n\n> +\n> +/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n> +#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n\nAnd why do you define this one here instead of in refs-be-files.c?\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271129","messageId":"561236E7.1050508@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-29-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 28/43] refs.c: add ref backend init function","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T08:37:59Z","receivedAt":"2015-10-05T08:37:59Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> The file backend doesn't need this function, but other backends might.\n> \n> Signed-off-by: David Turner \n> Signed-off-by: Ronnie Sahlberg \n> ---\n> refs-be-files.c | 1 +\n> refs.c | 4 +++-\n> refs.h | 4 +++-\n> 3 files changed, 7 insertions(+), 2 deletions(-)\n> \n> diff --git a/refs-be-files.c b/refs-be-files.c\n> index 37e244a..eaa74b6 100644\n> --- a/refs-be-files.c\n> +++ b/refs-be-files.c\n> @@ -3737,6 +3737,7 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n> struct ref_be refs_be_files = {\n> \tNULL,\n> \t\"files\",\n> +\tNULL,\n> \tfiles_transaction_begin,\n> \tfiles_transaction_update,\n> \tfiles_transaction_create,\n> diff --git a/refs.c b/refs.c\n> index 769574d..9ce10b7 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -19,13 +19,15 @@ struct ref_be *refs_backends = &refs_be_files;\n> /*\n> * This function is used to switch to an alternate backend.\n> */\n> -int set_refs_backend(const char *name)\n> +int set_refs_backend(const char *name, void *init_data)\n> {\n> \tstruct ref_be *be;\n> \n> \tfor (be = refs_backends; be; be = be->next)\n> \t\tif (!strcmp(be->name, name)) {\n> \t\t\tthe_refs_backend = be;\n> +\t\t\tif (be->init_backend)\n> +\t\t\t\tbe->init_backend(init_data);\n\nI don't like that this virtual function, alone among all of them\nintroduced so far, is allowed to be NULL. That seems non-obvious and\nsomething extra that devs have to remember.\n\nI think it would be better for the files backend to define a do-nothing\nfunction that can be stuck in this slot.\n\nIf you are opposed to that for some reason, then please at least add a\ncomment where this function pointer is added to `struct ref_be`, warning\nthat it can be NULL.\n\n> \t\t\treturn 0;\n> \t\t}\n> \treturn 1;\n> diff --git a/refs.h b/refs.h\n> index 0b407b2..0dc626e 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -586,6 +586,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,\n> \t\t\t void *policy_cb_data);\n> \n> /* refs backends */\n> +typedef void (*ref_backend_init_fn)(void *data);\n> typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n> typedef int (*ref_transaction_update_fn)(struct ref_transaction *transaction,\n> \t\tconst char *refname, const unsigned char *new_sha1,\n> @@ -641,6 +642,7 @@ typedef int (*for_each_reftype_fullpath_fn)(each_ref_fn fn, char *type,\n> struct ref_be {\n> \tstruct ref_be *next;\n> \tconst char *name;\n> +\tref_backend_init_fn init_backend;\n> \tref_transaction_begin_fn transaction_begin;\n> \tref_transaction_update_fn transaction_update;\n> \tref_transaction_create_fn transaction_create;\n> @@ -669,6 +671,6 @@ struct ref_be {\n> \n> \n> extern struct ref_be refs_be_files;\n> -int set_refs_backend(const char *name);\n> +int set_refs_backend(const char *name, void *init_data);\n> \n> #endif /* REFS_H */\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271130","messageId":"561237A5.4030603@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-31-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 30/43] refs-be-files.c: add method to expire reflogs","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T08:41:09Z","receivedAt":"2015-10-05T08:41:09Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 13 +++++++------\n> refs.c | 12 ++++++++++++\n> refs.h | 7 +++++++\n> 3 files changed, 26 insertions(+), 6 deletions(-)\n> [...]\n\nI think it would be natural to squash this into the previous patch.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271133","messageId":"56123B7E.4030808@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-38-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 37/43] refs: move some defines from refs-be-files.c to refs.h","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T08:57:34Z","receivedAt":"2015-10-05T08:57:34Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> This allows them to be used by other ref backends.\n> \n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 24 ------------------------\n> refs.h | 24 ++++++++++++++++++++++++\n> 2 files changed, 24 insertions(+), 24 deletions(-)\n> \n> diff --git a/refs-be-files.c b/refs-be-files.c\n> index 3bcfab3..2727943 100644\n> --- a/refs-be-files.c\n> +++ b/refs-be-files.c\n> @@ -14,30 +14,6 @@ struct ref_lock {\n> };\n> \n> /*\n> - * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken\n> - * refs (i.e., because the reference is about to be deleted anyway).\n> - */\n> -#define REF_DELETING\t0x02\n> [...]\n> diff --git a/refs.h b/refs.h\n> index 823983b..d8ae9af 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -193,6 +193,30 @@ struct ref_transaction {\n> #define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n> \n> /*\n> + * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken\n> + * refs (i.e., because the reference is about to be deleted anyway).\n> + */\n> +#define REF_DELETING\t0x02\n> [...]\n\nThe fact that these constants might be used by multiple reference\nbackends is not a reason to put them in the ref API's public interface.\nWe don't want random code using these contants, right?\n\nMaybe there should be a new file pair, refs-be-shared.{c,h}, for\n\"protected\" constants and functions for the internal use of the\nreference backends. In fact, maybe `struct ref_be` and the associated\ntypedefs could even be moved there.\n\nProbably some of the functions that you have made public earlier in this\nseries could also go into the \"protected\" area.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271135","messageId":"56123CE3.9070909@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-39-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T09:03:31Z","receivedAt":"2015-10-05T09:03:31Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Because HEAD and stash are per-worktree, other backends need to\n> go through the files backend to manage these refs and their reflogs.\n> \n> To enable this, we make some files backend functions public.\n\nI have a bad feeling about this change.\n\nNaively I would expect a reference backend that cannot handle its own\n(e.g.) stash to instantiate internally a files backend object and to\ndelegate stash-related calls to that object. That way neither class's\ninterface has to be changed.\n\nHere you are adding a separate interface to the files backend. That\nseems like a more complicated and less flexible design. But I'm open to\nbe persuaded otherwise...\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271136","messageId":"56123DAD.9010204@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-40-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 39/43] refs: break out a ref conflict check","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T09:06:53Z","receivedAt":"2015-10-05T09:06:53Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Create new function verify_no_descendants, to hold one of the ref\n> conflict checks used in verify_refname_available. Multiple backends\n> will need this function, so it goes in the common code.\n> \n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 33 ++++++++-------------------------\n> refs.c | 28 ++++++++++++++++++++++++++++\n> refs.h | 4 ++++\n> 3 files changed, 40 insertions(+), 25 deletions(-)\n> \n> diff --git a/refs-be-files.c b/refs-be-files.c\n> index 943604c..8815111 100644\n> --- a/refs-be-files.c\n> +++ b/refs-be-files.c\n> @@ -753,6 +753,7 @@ static int verify_refname_available_dir(const char *refname,\n> \t\t\t\t\tstruct strbuf *err)\n> {\n> \tconst char *slash;\n> +\tconst char *extra_refname;\n> \tint pos;\n> \tstruct strbuf dirname = STRBUF_INIT;\n> \tint ret = -1;\n> @@ -858,33 +859,15 @@ static int verify_refname_available_dir(const char *refname,\n> \t\t}\n> \t}\n> \n> -\tif (extras) {\n> -\t\t/*\n> -\t\t * Check for entries in extras that start with\n> -\t\t * \"$refname/\". We do that by looking for the place\n> -\t\t * where \"$refname/\" would be inserted in extras. If\n> -\t\t * there is an entry at that position that starts with\n> -\t\t * \"$refname/\" and is not in skip, then we have a\n> -\t\t * conflict.\n> -\t\t */\n> -\t\tfor (pos = string_list_find_insert_index(extras, dirname.buf, 0);\n> -\t\t pos < extras->nr; pos++) {\n> -\t\t\tconst char *extra_refname = extras->items[pos].string;\n> -\n> -\t\t\tif (!starts_with(extra_refname, dirname.buf))\n> -\t\t\t\tbreak;\n> -\n> -\t\t\tif (!skip || !string_list_has_string(skip, extra_refname)) {\n> -\t\t\t\tstrbuf_addf(err, \"cannot process '%s' and '%s' at the same time\",\n> -\t\t\t\t\t refname, extra_refname);\n> -\t\t\t\tgoto cleanup;\n> -\t\t\t}\n> -\t\t}\n> +\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n> +\tif (extra_refname) {\n> +\t\tstrbuf_addf(err,\n> +\t\t\t \"cannot process '%s' and '%s' at the same time\",\n> +\t\t\t refname, extra_refname);\n> +\t} else {\n> +\t\tret = 0;\n> \t}\n> \n> -\t/* No conflicts were found */\n> -\tret = 0;\n> -\n> cleanup:\n> \tstrbuf_release(&dirname);\n> \treturn ret;\n> diff --git a/refs.c b/refs.c\n> index 1c2dd79..17a364a 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -994,6 +994,34 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)\n> \treturn PEEL_PEELED;\n> }\n> \n> +const char *find_descendant_ref(const char *refname,\n> +\t\t\t\tconst struct string_list *extras,\n> +\t\t\t\tconst struct string_list *skip)\n> +{\n> +\tint pos;\n> +\tif (!extras)\n> +\t\treturn NULL;\n> +\n> +\t/*\n> +\t * Check for entries in extras that start with \"$refname/\". We\n> +\t * do that by looking for the place where \"$refname/\" would be\n> +\t * inserted in extras. If there is an entry at that position\n> +\t * that starts with \"$refname/\" and is not in skip, then we\n> +\t * have a conflict.\n> +\t */\n\nWould you please turn this comment (or something like it) into a\ndocstring for this function in refs.h?\n\n> +\tfor (pos = string_list_find_insert_index(extras, refname, 0);\n> +\t pos < extras->nr; pos++) {\n> +\t\tconst char *extra_refname = extras->items[pos].string;\n> +\n> +\t\tif (!starts_with(extra_refname, refname))\n> +\t\t\tbreak;\n> +\n> +\t\tif (!skip || !string_list_has_string(skip, extra_refname))\n> +\t\t\treturn extra_refname;\n> +\t}\n> +\treturn NULL;\n> +}\n> +\n> /* backend functions */\n> int refs_initdb(struct strbuf *err, int shared)\n> {\n> diff --git a/refs.h b/refs.h\n> index da29232..cf1780e 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -571,6 +571,10 @@ enum ref_type ref_type(const char *refname);\n> \n> int copy_reflog_msg(char *buf, const char *msg);\n> \n> +const char *find_descendant_ref(const char *refname,\n> +\t\t\t\tconst struct string_list *extras,\n> +\t\t\t\tconst struct string_list *skip);\n> +\n> int files_log_ref_write(const char *refname, const unsigned char *old_sha1,\n> \t\t\tconst unsigned char *new_sha1, const char *msg,\n> \t\t\tint flags, struct strbuf *err);\n> \n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271137","messageId":"5612439E.4080200@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-41-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T09:32:14Z","receivedAt":"2015-10-05T09:32:14Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Add a new option, --refs-backend-type, to allow the ref backend type to\n> be set on new clones.\n> \n> Submodules must use the same ref backend as the parent repository, so\n> we also pass the --refs-backend-type option option when cloning\n> submodules.\n\nIf I'm reading this correctly, you propose to add a new configuration\nsetting,\n\n core.refs-backend-type\n\nto record which reference backend the repository is using.\n\n(By the way, please document any new config settings in\nDocumentation/config.txt.)\n\nThis does not provide adequate backwards compatibility. Suppose I create\na repository with a DB backend, then later try to access the repository\nusing an older version of Git (or JGit or libgit2 or ...) that doesn't\nknow about this configuration setting. It will think it is a normal\nrepository that only has a couple of references (HEAD, stash, etc). If I\nrun `git gc` using the older version of git, *poof* all of the objects\nreferred to by other branches will be garbage-collected.\n\nTherefore, I don't think this can be merged without a bump to\ncore.repositoryformatversion. Such a bump will tell well-behaved older\nGit clients keep their hands off the repository. (Of course repositories\nthat use the files backend can continue using\ncore.repositoryformatversion 0.)\n\nI thought Peff proposed a way to do such a bump, including a way to\nextend repositories one by one with new features. But that was something\nthat we were chatting about off-list.\n\nI haven't reviewed the actual code in this patch yet but I wanted to get\nthe above comment on your radar.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271140","messageId":"5612653F.5070502@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-41-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T11:55:43Z","receivedAt":"2015-10-05T11:55:43Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Add a new option, --refs-backend-type, to allow the ref backend type to\n> be set on new clones.\n> \n> Submodules must use the same ref backend as the parent repository, so\n> we also pass the --refs-backend-type option option when cloning\n> submodules.\n> \n> Signed-off-by: David Turner \n> ---\n> Documentation/git-clone.txt | 4 ++++\n> builtin/clone.c | 27 +++++++++++++++++++++++++--\n> builtin/submodule--helper.c | 5 ++++-\n> cache.h | 1 +\n> refs.c | 2 ++\n> 5 files changed, 36 insertions(+), 3 deletions(-)\n> \n> [...]\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 3e14491..d489a87 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> [...]\n> @@ -744,6 +762,11 @@ static void write_config(struct string_list *config)\n> \t\t\t\t\t write_one_config, NULL) < 0)\n> \t\t\tdie(\"unable to write parameters to config file\");\n> \t}\n> +\n> +\tif (refs_backend_type &&\n> +\t write_one_config(\"core.refs-backend-type\",\n> +\t\t\t refs_backend_type, NULL) < 0)\n> +\t\t\tdie(\"unable to write backend parameter to config file\");\n> }\n\nOur config variable names don't use separated-with-hyphens but rather\nwordsruntogether (though in documentation this would be written\ncore.refsBackendType, because the actual names are case-insensitive).\n\n> [...]\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271142","messageId":"56129B77.1030409@alum.mit.edu","threadId":"40445","inReplyTo":"1443477738-32023-43-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-05T15:47:03Z","receivedAt":"2015-10-05T15:47:03Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 09/29/2015 12:02 AM, David Turner wrote:\n> Add a database backend for refs using LMDB. This backend runs git\n> for-each-ref about 30% faster than the files backend with fully-packed\n> refs on a repo with ~120k refs. It's also about 4x faster than using\n> fully-unpacked refs. In addition, and perhaps more importantly , it\n> avoids case-conflict issues on OS X.\n> \n> LMDB has a few features that make it suitable for usage in git:\n> \n> 1. It is relatively lightweight; it requires only one header file, and\n> the library code takes under 64k at runtime.\n> \n> 2. It is well-tested: it's been used in OpenLDAP for years.\n> \n> 3. It's very fast. LMDB's benchmarks show that it is among\n> the fastest key-value stores.\n> \n> 4. It has a relatively simple concurrency story; readers don't\n> block writers and writers don't block readers.\n\nIt would be great if you would go into more detail about this point in\nthe permanent technical documentation\n(Documentation/technical/refs-be-lmdb.txt): how exactly do readers &\nwriters interact with each others, and how do writers interact with\nother writers?\n\nI think you have said before that if one writer holds the write lock on\nthe DB, then other writers fail immediately. Is that correct? If so, is\nthat something that can be adjusted? I think it would be preferable for\nthe second writer to retry acquiring the write lock for a little while\nwith a timeout (as we now do when trying to acquire the packed-refs\nlock). Otherwise you could have the unhappy situation that somebody\nspends a long time pushing a packfile to a server, only to have the\nreference update be rejected at the last moment due to a lock conflict\nwith another process that was touching completely different references.\nWe already do before/after consistency checks when updating references,\nso you wouldn't even have to add such code in the backend yourself.\n\n> Ronnie Sahlberg's original version of this patchset used tdb. The\n> major disadvantage of tdb is that tdb is hard to build on OS X. It's\n> also not in homebrew. So lmdb seemed simpler.\n> \n> To test this backend's correctness, I hacked test-lib.sh and\n> test-lib-functions.sh to run all tests under the refs backend. Dozens\n> of tests use manual ref/reflog reading/writing, or create submodules\n> without passing --refs-backend-type to git init. If those tests are\n> changed to use the update-ref machinery or test-refs-be-db (or, in the\n> case of packed-refs, corrupt refs, and dumb fetch tests, are skipped),\n> the only remaining failing tests are the git-new-workdir tests and the\n> gitweb tests.\n> \n> Signed-off-by: David Turner \n> ---\n> .gitignore | 1 +\n> Documentation/git-clone.txt | 2 +-\n> Documentation/git-init-db.txt | 2 +-\n> Documentation/git-init.txt | 6 +\n> Documentation/technical/refs-be-lmdb.txt | 39 +\n> Makefile | 12 +\n> builtin/init-db.c | 13 +\n> config.c | 27 +\n> configure.ac | 33 +\n> contrib/workdir/git-new-workdir | 2 +\n> environment.c | 1 +\n> refs-be-lmdb.c | 2003 ++++++++++++++++++++++++++++++\n> refs.h | 10 +\n> setup.c | 28 +-\n> test-refs-be-lmdb.c | 68 +\n> 15 files changed, 2239 insertions(+), 8 deletions(-)\n> create mode 100644 Documentation/technical/refs-be-lmdb.txt\n> create mode 100644 refs-be-lmdb.c\n> create mode 100644 test-refs-be-lmdb.c\n> \n> diff --git a/.gitignore b/.gitignore\n> index 1c2f832..7decc2f 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -199,6 +199,7 @@\n> /test-path-utils\n> /test-prio-queue\n> /test-read-cache\n> +/test-refs-be-lmdb\n> /test-regex\n> /test-revision-walking\n> /test-run-command\n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index d7a4cb0..6a4cf28 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -219,7 +219,7 @@ objects from the source repository into a pack in the cloned repository.\n> \n> --refs-backend-type=::\n> \tType of refs backend. Default is to use the original files based\n> -\tbackend.\n> +\tbackend. Set to \"lmdb\" to activate the lmdb database backend.\n> \n> ::\n> \tThe (possibly remote) repository to clone from. See the\n> diff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt\n> index 648a6cd..72fbd71 100644\n> --- a/Documentation/git-init-db.txt\n> +++ b/Documentation/git-init-db.txt\n> @@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository\n> SYNOPSIS\n> --------\n> [verse]\n> -'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]]\n> +'git init-db' [-q | --quiet] [--bare] [--template=] [--separate-git-dir ] [--shared[=]] [--refs-backend-type=]\n> \n> \n> DESCRIPTION\n> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\n> index 8174d27..d352788 100644\n> --- a/Documentation/git-init.txt\n> +++ b/Documentation/git-init.txt\n> @@ -12,6 +12,7 @@ SYNOPSIS\n> 'git init' [-q | --quiet] [--bare] [--template=]\n> \t [--separate-git-dir ]\n> \t [--shared[=]] [directory]\n> +\t [--refs-backend-type=]\n> \n> \n> DESCRIPTION\n> @@ -113,6 +114,11 @@ does not exist, it will be created.\n> \n> --\n> \n> +--refs-backend-type=::\n> +Type of refs backend. Default is to use the original \"files\" backend,\n> +which stores ref data in files in .git/refs and .git/packed-refs. Set\n> +to \"lmdb\" to activate the lmdb database backend.\n> +\n> TEMPLATE DIRECTORY\n> ------------------\n> \n> diff --git a/Documentation/technical/refs-be-lmdb.txt b/Documentation/technical/refs-be-lmdb.txt\n> new file mode 100644\n> index 0000000..98c13ec\n> --- /dev/null\n> +++ b/Documentation/technical/refs-be-lmdb.txt\n> @@ -0,0 +1,39 @@\n> +Notes on the LMDB refs backend\n> +==============================\n> +\n> +Design:\n> +------\n> +\n> +Refs and reflogs are stored in a lmdb database in .git/refdb. All\n> +keys and values are \\0-terminated.\n> +\n> +Keys for refs are the name of the ref (e.g. refs/heads/master).\n> +Values are the value of the ref\n> +(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n\nPlease document more unambiguously whether the value is stored as\n40-char hexadecimal or 20-byte binary.\n\nDo you store \"peeled\" reference values for tags, as is done in\npacked-refs? I think that is an important optimization.\n\n> +All per-worktree refs (refs/bisect/* and HEAD) are store using\n> +the traditional files-based backend.\n> +\n> +Reflogs are stored as a series of database entries.\n> +\n> +For an empty reflog, there is a \"header\" entry to show that a reflog\n> +exists.\n\nWhat key is used for the \"header\" entry? Does it have a value?\n\n> + For non-empty reflogs, there is one entry per logged ref\n> +update. The key format is logs/[refname]\\0[timestamp]. The timestamp\n> +is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n> +This means that reflog entries are chronologically ordered. Because\n> +LMDB is a btree database, we can efficiently iterate over these keys.\n\nCurrently we discard the reflog for a reference when the reference is\ndeleted. This is mostly due to a limitation of the filesystem-based\nstorage scheme--otherwise the reflog left over from a deleted reference\ncould prevent the creation of a reflog for another reference whose name\noverlaps with it (in the sense of is_refname_available()).\n\nIt is a pretty nasty limitation because it is one of the few ways that\nyour actions can cause Git to lose data in a way that it is hard to\nrecover from.\n\nHave you thought about removing this limitation in the lbdb backend?\n\n> +Reflog values are in the same format as the original files-based\n> +reflog.\n> +\n> +Weaknesses:\n> +-----------\n> +\n> +The reflog format is somewhat inefficient: a binary format could store\n> +reflog date/time information in somewhat less space.\n> +\n> +The rsync and file:// transports don't work yet, because they\n> +don't use the refs API.\n\nDo they fail gracefully?\n\n> +git new-workdir is incompatible with the lmdb backend. Fortunately,\n> +git new-workdir is deprecated, and worktrees work fine.\n> diff --git a/Makefile b/Makefile\n> index 43ceab0..e96f84b 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1036,6 +1036,17 @@ ifdef USE_LIBPCRE\n> \tEXTLIBS += -lpcre\n> endif\n> \n> +ifdef USE_LIBLMDB\n> +\tBASIC_CFLAGS += -DUSE_LIBLMDB\n> +\tifdef LIBLMDBDIR\n> +\t\tBASIC_CFLAGS += -I$(LIBLMDBDIR)/include\n> +\t\tEXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)\n> +\tendif\n> +\tEXTLIBS += -llmdb\n> +\tLIB_OBJS += refs-be-lmdb.o\n> +\tTEST_PROGRAMS_NEED_X += test-refs-be-lmdb\n> +endif\n> +\n> ifdef HAVE_ALLOCA_H\n> \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n> endif\n> @@ -2122,6 +2133,7 @@ GIT-BUILD-OPTIONS: FORCE\n> \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n> \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n> \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n> +\t@echo USE_LIBLMDB=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBLMDB)))'\\' >>$@+\n> \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n> \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n> \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 504a2dc..89f2c05 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -203,6 +203,17 @@ static int create_default_files(const char *template_path)\n> \t\tadjust_shared_perm(get_git_dir());\n> \t}\n> \n> +\tif (refs_backend_type) {\n> +\t\tstruct refdb_config_data config_data = {NULL};\n> +\t\tgit_config_set(\"core.refs-backend-type\", refs_backend_type);\n> +\t\tconfig_data.refs_backend_type = refs_backend_type;\n> +\t\tconfig_data.refs_base = get_git_dir();\n> +#ifdef USE_LIBLMDB\n> +\t\tregister_refs_backend(&refs_be_lmdb);\n> +#endif\n> +\t\tset_refs_backend(refs_backend_type, &config_data);\n> +\t}\n> +\n> \tif (refs_initdb(&err, shared_repository))\n> \t\tdie(\"failed to set up refs db: %s\", err.buf);\n> \n> @@ -468,6 +479,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n> \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n> \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n> \t\t\t N_(\"separate git dir from working tree\")),\n> +\t\tOPT_STRING(0, \"refs-backend-type\", &refs_backend_type,\n> +\t\t\t N_(\"name\"), N_(\"name of backend type to use\")),\n> \t\tOPT_END()\n> \t};\n> \n> diff --git a/config.c b/config.c\n> index 248a21a..9ce2ed2 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -10,6 +10,7 @@\n> #include \"exec_cmd.h\"\n> #include \"strbuf.h\"\n> #include \"quote.h\"\n> +#include \"refs.h\"\n> #include \"hashmap.h\"\n> #include \"string-list.h\"\n> #include \"utf8.h\"\n> @@ -1207,6 +1208,32 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)\n> \t}\n> \n> \tif (repo_config && !access_or_die(repo_config, R_OK, 0)) {\n> +\t\tstruct refdb_config_data refdb_data = {NULL};\n> +\t\tchar *repo_config_copy;\n> +\n> +\t\t/*\n> +\t\t * make sure we always read the backend config from the\n> +\t\t * core section on startup\n> +\t\t */\n> +\t\tret += git_config_from_file(refdb_config, repo_config,\n> +\t\t\t\t\t &refdb_data);\n> +\n> +\t\trepo_config_copy = xstrdup(repo_config);\n> +\t\trefdb_data.refs_base = strdup(dirname(repo_config_copy));\n> +\t\tfree(repo_config_copy);\n> +\n> +\t\tif (refdb_data.refs_backend_type &&\n> +\t\t !strcmp(refdb_data.refs_backend_type, \"lmdb\")) {\n> +\n> +#ifdef USE_LIBLMDB\n> +\t\t\trefs_backend_type = refdb_data.refs_backend_type;\n> +\t\t\tregister_refs_backend(&refs_be_lmdb);\n> +\t\t\tset_refs_backend(refs_backend_type, &refdb_data);\n> +#else\n> +\t\t\tdie(\"Git was not built with USE_LIBLMDB, so the db refs backend is not available\");\n\ns/db/lmdb/\n\nI'm somewhat surprised that you only register the lmdb backend if it is\nused in the main repo. I would expect the backend to be registered\nunconditionally on startup. The cost is trivial, isn't it?\n\nIt appears that you recognize \"lmdb\" but you treat a missing value *or\nany other value* like \"files\". I think you should be more conservative\nhere and die() if you see a backend type that you don't recognize.\n\n> +#endif\n> +\t\t}\n> +\n> \t\tret += git_config_from_file(fn, repo_config, data);\n> \t\tfound += 1;\n> \t}\n> diff --git a/configure.ac b/configure.ac\n> index 63b7244..ea8e93d 100644\n> --- a/configure.ac\n> +++ b/configure.ac\n> @@ -271,6 +271,24 @@ AS_HELP_STRING([], [ARG can be also prefix for libpcre library and hea\n> dnl it yet.\n> \tGIT_CONF_SUBST([LIBPCREDIR])\n> fi)\n> +\n> +USE_LIBLMDB=YesPlease\n> +AC_ARG_WITH(liblmdb,\n> +AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])\n> +AS_HELP_STRING([], [ARG can be also prefix for liblmdb library and headers]),\n> + if test \"$withval\" = \"no\"; then\n> +\tUSE_LIBLMDB=\n> + elif test \"$withval\" = \"yes\"; then\n> +\tUSE_LIBLMDB=YesPlease\n> + else\n> +\tUSE_LIBLMDB=YesPlease\n> +\tLIBLMDBDIR=$withval\n> +\tAC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])\n> + dnl USE_LIBLMDB can still be modified below, so don't substitute\n> + dnl it yet.\n> +\tGIT_CONF_SUBST([LIBLMDBDIR])\n> + fi)\n> +\n> #\n> # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n> AC_FUNC_ALLOCA\n> @@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])\n> \n> fi\n> \n> +if test -n \"$USE_LIBLMDB\"; then\n> +\n> +GIT_STASH_FLAGS($LIBLMDBDIR)\n> +\n> +AC_CHECK_LIB([lmdb], [mdb_env_open],\n> +[USE_LIBLMDB=YesPlease],\n> +[USE_LIBLMDB=])\n> +\n> +GIT_UNSTASH_FLAGS($LIBLMDBDIR)\n> +\n> +GIT_CONF_SUBST([USE_LIBLMDB])\n> +\n> +fi\n> +\n> +\n> #\n> # Define NO_CURL if you do not have libcurl installed. git-http-pull and\n> # git-http-push are not built, and you cannot use http:// and https://\n> diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\n> index 888c34a..9761a1a 100755\n> --- a/contrib/workdir/git-new-workdir\n> +++ b/contrib/workdir/git-new-workdir\n> @@ -28,6 +28,8 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n> git rev-parse --git-dir 2>/dev/null) ||\n> die \"Not a git repository: \\\"$orig_git\\\"\"\n> \n> +test -d $git_dir/refdb && die \"git-new-workdir is incompatible with the refs db backend\"\n> +\n\nIt would be safer to check $(git config core.refs-backend-type) to avoid\ndepending on an implementation detail of the LMDB backend (and as a bit\nof safety against any backends that might be added in the future).\n\n> case \"$git_dir\" in\n> .git)\n> \tgit_dir=\"$orig_git/.git\"\n> diff --git a/environment.c b/environment.c\n> index c5b65f5..6a2f639 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -65,6 +65,7 @@ int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> struct startup_info *startup_info;\n> unsigned long pack_size_limit_cfg;\n> +const char *refs_backend_type;\n> \n> #ifndef PROTECT_HFS_DEFAULT\n> #define PROTECT_HFS_DEFAULT 0\n> diff --git a/refs-be-lmdb.c b/refs-be-lmdb.c\n> new file mode 100644\n> index 0000000..99cbd29\n> --- /dev/null\n> +++ b/refs-be-lmdb.c\n> @@ -0,0 +1,2003 @@\n> +/*\n> + * This file implements a lmdb backend for refs.\n> + *\n> + * The design of this backend relies on lmdb's write lock -- that is, any\n> + * write transaction blocks all other writers. Thus, as soon as a ref\n> + * transaction is opened, we know that any values we read won't\n> + * change out from under us, and we have a fully-consistent view of the\n> + * database.\n> + *\n> + * We store the content of refs including the trailing \\0 so that\n> + * standard C string functions can handle them. Just like struct\n> + * strbuf.\n> + */\n> +#include \n> +#include \n> +#include \"cache.h\"\n> +#include \"object.h\"\n> +#include \"refs.h\"\n> +#include \"tag.h\"\n> +#include \"lockfile.h\"\n> +#include \"run-command.h\"\n> +\n> +static struct trace_key db_trace = TRACE_KEY_INIT(LMDB);\n> +\n> +static MDB_env *env;\n> +\n> +static char *db_path;\n> +\n> +struct lmdb_transaction_info {\n> +\tMDB_txn *txn;\n> +\tMDB_dbi dbi;\n> +\tconst char *submodule;\n> +};\n> +\n> +struct lmdb_transaction {\n> +\tstruct ref_transaction base;\n> +\tMDB_cursor *cursor;\n> +\tstruct lmdb_transaction_info info;\n> +\tstruct hashmap updated_refs;\n> +\tunsigned int flags;\n> +};\n> +\n> +static struct lmdb_transaction transaction = {{}, NULL};\n> +\n> +struct ref_update {\n> +\tstruct hashmap_entry ent; /* must be first */\n> +\tsize_t len;\n> +\tchar refname[FLEX_ARRAY];\n> +};\n> +\n> +static char *get_refdb_path(const char *base)\n> +{\n> +\tstruct strbuf path_buf = STRBUF_INIT;\n> +\tstrbuf_addf(&path_buf, \"%s/refdb\", base);\n> +\treturn strbuf_detach(&path_buf, NULL);\n> +}\n> +\n> +static int in_write_transaction(void)\n> +{\n> +\treturn transaction.info.txn && !(transaction.flags & MDB_RDONLY);\n> +}\n> +\n> +static void init_env(MDB_env **env, const char *path)\n> +{\n> +\tint ret;\n> +\tif (*env)\n> +\t\treturn;\n> +\n> +\tif ((ret = mdb_env_create(env)) != MDB_SUCCESS)\n> +\t\tdie(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n\nThe LMDB docs seem to say that their functions \"return a non-zero error\nvalue on failure and 0 on success\". So can we avoid writing MDB_SUCCESS\neverywhere and just compare to zero?\n\nAlso, the Git project CodingGuidelines say to avoid assignments in the\ncondition of an \"if\" statement.\n\nSo all in all I think these should be written like\n\n ret = mdb_env_create(env);\n if (ret)\n die(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n\n> +\tif ((ret = mdb_env_set_maxreaders(*env, 1000)) != MDB_SUCCESS)\n> +\t\tdie(\"BUG: mdb_env_set_maxreaders failed: %s\", mdb_strerror(ret));\n> +\tif ((ret = mdb_env_set_mapsize(*env, (1<<30))) != MDB_SUCCESS)\n> +\t\tdie(\"BUG: mdb_set_mapsize failed: %s\", mdb_strerror(ret));\n> +\tif ((ret = mdb_env_open(*env, path, 0 , 0664)) != MDB_SUCCESS)\n> +\t\tdie(\"BUG: mdb_env_open (%s) failed: %s\", path, mdb_strerror(ret));\n> +}\n> +\n> +static int lmdb_initdb(struct strbuf *err, int shared)\n> +{\n> +\t/* To create a db, all we need to do is make a directory for\n> +\t it to live in; lmdb will do the rest. */\n> +\n> +\tassert(db_path);\n> +\tif (mkdir(db_path, 0775)) {\n> +\t\tif (errno != EEXIST) {\n> +\t\t\tstrbuf_addf(err, \"%s\", strerror(errno));\n> +\t\t\treturn -1;\n> +\t\t}\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static void lmdb_init_backend(struct refdb_config_data *data)\n> +{\n> +\tif (db_path)\n> +\t\treturn;\n> +\n> +\tdb_path = xstrdup(real_path(get_refdb_path(data->refs_base)));\n> +\ttrace_printf_key(&db_trace, \"Init backend\\n\");\n> +}\n> +\n> +int ref_update_cmp(const void *entry, const void *entry_or_key, const void *keydata)\n\nThe usual convention for a \"cmp\" function is that it returns <0, =0, or\n>0 depending on the relative ordering of its arguments. This one can\nonly be used for equality/inequality. Please add a comment warning the\nreader of this (and maybe rename it, e.g., to ref_update_not_equal(), to\nfurther reduce the chance of confusion).\n\nAside: the hashmap_init() documentation is very confusing. Based on the\ndocumentation alone, I would have thought that its `equals_function`\nshould return true if the two arguments are *equal*.\n\n> +{\n> +\n> +\tconst struct ref_update *existing = entry;\n> +\tconst struct ref_update *incoming = entry_or_key;\n> +\n> +\treturn existing->len != incoming->len ||\n> +\t\tmemcmp(existing->refname, incoming->refname, existing->len);\n> +}\n> +\n> +static void mdb_cursor_open_or_die(struct lmdb_transaction_info *info,\n> +\t\t\t\t MDB_cursor **cursor)\n> +{\n> +\tint ret = mdb_cursor_open(info->txn, info->dbi, cursor);\n> +\tif (ret)\n> +\t\tdie(\"mdb_cursor_open failed: %s\", mdb_strerror(ret));\n> +}\n> +\n> +static void submodule_path(struct strbuf *sb, const char *submodule,\n> +\t\t\t const char *refname)\n> +{\n> +\tif (submodule)\n> +\t\tstrbuf_git_path_submodule(sb, submodule, \"%s\", refname);\n> +\telse\n> +\t\tstrbuf_git_path(sb, \"%s\", refname);\n> +}\n> +\n> +static int read_per_worktree_ref(const char *submodule, const char *refname,\n> +\t\t\t\t struct MDB_val *val)\n> +{\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> +\n> +\tsubmodule_path(&path, submodule, refname);\n> +\n> +\tif (strbuf_read_file(&sb, path.buf, 200) < 0) {\n> +\t\tstrbuf_release(&sb);\n> +\t\tstrbuf_release(&path);\n> +\t\tif (errno == ENOENT)\n> +\t\t\treturn MDB_NOTFOUND;\n> +\t\treturn -1;\n> +\t}\n> +\tstrbuf_setlen(&sb, sb.len - 1);\n> +\n> +\tval->mv_data = sb.buf;\n> +\tval->mv_size = sb.len + 1;\n> +\n> +\tstrbuf_detach(&sb, NULL);\n> +\tstrbuf_release(&path);\n> +\treturn 0;\n> +}\n> +\n> +static void write_per_worktree_ref(const char *submodule, const char *refname,\n> +\t\t\t\t MDB_val *val)\n> +{\n> +\tstatic struct lock_file lock;\n> +\tint fd;\n> +\tint len = val->mv_size - 1;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\n> +\tsubmodule_path(&path, submodule, refname);\n> +\tsafe_create_leading_directories(path.buf);\n> +\n> +\tfd = hold_lock_file_for_update(&lock, path.buf, LOCK_DIE_ON_ERROR);\n> +\tstrbuf_release(&path);\n> +\n> +\tif (write_in_full(fd, val->mv_data, len) != len ||\n> +\t write_in_full(fd, \"\\n\", 1) != 1)\n> +\t\tdie_errno(\"failed to write new HEAD\");\n> +\n> +\tif (commit_lock_file(&lock))\n> +\t\tdie_errno(\"failed to write new HEAD\");\n> +}\n> +\n> +static int del_per_worktree_ref(const char *submodule, const char *refname,\n> +\t\t\t\tMDB_val *val)\n> +{\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tint result;\n> +\n> +\t/*\n> +\t * Returning deleted ref data is not yet implemented, but no\n> +\t * callers need it.\n> +\t */\n> +\tassert(val == NULL);\n> +\n> +\tsubmodule_path(&path, submodule, refname);\n> +\n> +\tresult = unlink(path.buf);\n> +\tstrbuf_release(&path);\n> +\tif (result) {\n> +\t\tif (errno == ENOENT)\n> +\t\t\treturn 0;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +static int mdb_get_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val)\n> +{\n> +\tint ret;\n> +\n> +\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n> +\t\treturn read_per_worktree_ref(info->submodule, key->mv_data, val);\n> +\n> +\tret = mdb_get(info->txn, info->dbi, key, val);\n> +\tif (ret) {\n> +\t\tif (ret != MDB_NOTFOUND)\n> +\t\t\tdie(\"mdb_get failed: %s\", mdb_strerror(ret));\n> +\t\treturn ret;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static int mdb_del_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val)\n> +{\n> +\tint ret;\n> +\n> +\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL)\n> +\t\treturn del_per_worktree_ref(info->submodule, key->mv_data, val);\n> +\n> +\tret = mdb_del(info->txn, info->dbi, key, val);\n> +\tif (ret) {\n> +\t\tif (ret != MDB_NOTFOUND)\n> +\t\t\tdie(\"mdb_del failed: %s\", mdb_strerror(ret));\n> +\t\treturn ret;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static void mdb_put_or_die(struct lmdb_transaction_info *info, MDB_val *key, MDB_val *val, int mode)\n> +{\n> +\tint ret;\n> +\n> +\tif (ref_type(key->mv_data) != REF_TYPE_NORMAL) {\n> +\t\twrite_per_worktree_ref(info->submodule, key->mv_data, val);\n> +\t\treturn;\n> +\t}\n> +\tassert(val->mv_size == 0 || ((char *)val->mv_data)[val->mv_size - 1] == 0);\n> +\n> +\tret = mdb_put(info->txn, info->dbi, key, val, mode);\n> +\tif (ret) {\n> +\t\tif (ret == MDB_BAD_VALSIZE)\n> +\t\t\tdie(\"Ref name %s too long (max size is %d)\",\n> +\t\t\t (const char *)key->mv_data,\n> +\t\t\t mdb_env_get_maxkeysize(env));\n> +\t\telse\n> +\t\t\tdie(\"mdb_put failed: %s\", mdb_strerror(ret));\n> +\t}\n> +}\n> +\n> +static int mdb_cursor_get_or_die(MDB_cursor *cursor, MDB_val *key, MDB_val *val, int mode)\n> +{\n> +\tint ret;\n> +\n> +\tret = mdb_cursor_get(cursor, key, val, mode);\n> +\tif (ret) {\n> +\t\tif (ret != MDB_NOTFOUND)\n> +\t\t\tdie(\"mdb_cursor_get failed: %s\", mdb_strerror(ret));\n> +\t\treturn ret;\n> +\t}\n> +\tassert(((char *)val->mv_data)[val->mv_size - 1] == 0);\n> +\treturn 0;\n> +}\n> +\n> +static int mdb_cursor_del_or_die(MDB_cursor *cursor, int flags)\n> +{\n> +\tint ret = mdb_cursor_del(cursor, flags);\n> +\tif (ret) {\n> +\t\tif (ret != MDB_NOTFOUND)\n> +\t\t\tdie(\"mdb_cursor_del failed: %s\", mdb_strerror(ret));\n> +\t\treturn ret;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +/*\n> + * Begin a transaction. Because only one transaction per thread is\n> + * permitted, we use a global transaction object. If a read-write\n> + * transaction is presently already in-progress, and a read-only\n> + * transaction is requested, the read-write transaction will be\n> + * returned instead. If a read-write transaction is requested and a\n> + * read-only transaction is open, the read-only transaction will be\n> + * closed.\n> + *\n> + * It is a bug to request a read-write transaction during another\n> + * read-write transaction.\n> + *\n> + * As a result, it is unsafe to retain read-only transactions past the\n> + * point where a read-write transaction might be needed. For\n> + * instance, any call that has callbacks outside this module must\n> + * conclude all of its reads from the database before calling those\n> + * callbacks, or must reacquire the transaction after its callbacks\n> + * are completed.\n> + */\n> +int lmdb_transaction_begin_flags(struct strbuf *err, unsigned int flags)\n> +{\n> +\tint ret;\n> +\tMDB_txn *txn;\n> +\tstatic int last_commands_run = 0;\n> +\tint force_restart = 0;\n> +\n> +\tinit_env(&env, db_path);\n> +\n> +\tif (total_commands_run != last_commands_run) {\n> +\t\t/*\n> +\t\t * Since each transaction sees a consistent view of\n> +\t\t * the db, downstream processes that write the db\n> +\t\t * won't be seen in this transaction. We don't know\n> +\t\t * whether any given downstream process has made any\n> +\t\t * writes, so if there have been any downstream processes,\n> +\t\t * we had better reopen the transaction.\n> +\t\t */\n\nIs it possible to tell from the database whether there has been a write\nsince the current read transaction was started? It seems like this would\nbe a more reliable method, given that an unrelated process might have\nupdated references while this command was running, or a long-running\nbackground process (like Duy's background `git gc --auto`) might still\ncontinue running even after the original command exits.\n\nOf course in the end checks like this can never 100% prevent races. The\nonly way, it would seem, would be to check-and-set during a single write\ntransaction that holds a write lock on the DB.\n\nThat's exactly what we do for file-based references: ideally, the only\nallowed changes are compare-and-set operations that are made atomic by\nholding a lock while it is occurring (one or more loose reference\nlockfiles and/or the packed-refs lockfile). That is why our ref_updates\nhave old_sha1 and new_sha1, and old_sha1 is always used for operations\nthat start before a lock is held.\n\nSo I guess my question is this: to what extent are you relying on this\ntotal_commands_run mechanism for correctness, vs. just using it to avoid\nusing reference values that are *really* old?\n\nIt seems to me that a very important feature of Git is the following:\n\n Two processes *must not* block each other (except possibly\n for a very brief time) if they are updating disjoint sets\n of references.\n\nWith the file-backed reference backend, this feature is accomplished via\nthe following mechanisms:\n\n* Locks are only held for very short periods of time, and never when\n calling other processes.\n\n* Locks usually only apply to single references (loose references).\n Sometimes the packed-refs file has to be locked, but in such cases\n a second writer retries the lock acquisition for long enough that\n it should usually succeed.\n\n* If a reference value was checked before a lock was held, then it\n is always re-checked after acquiring the lock to make sure that\n another process didn't change it in the meantime. (If the\n reference *was* changed from its expected value, it is ok to\n die().) This is necessary to preserve safety even if a lock is\n not held for the duration of a command.\n\nIn particular, locks are only held for the duration of the call to\nref_transaction_commit(), *not* while building up a transaction using\nref_transaction_update() etc.\n\nI am afraid that the LMDB backend as currently implemented will be more\nsubject to locking conflicts because the LMDB locks are broader in both\ntime and space:\n\n1. You are holding the LMDB transaction open for longer (during the\n whole time that the ref_transaction is being built up). This is\n worse simply because it is a longer time. But it is also worse\n because the process that is building up the transaction might be\n doing other things while holding the write lock. Have you checked\n whether any callers do things that could change references (for\n example, invoke user hooks) between the calls to\n ref_transaction_begin() and ref_transaction_commit()? With the\n files backend this isn't an issue because the lock isn't acquired\n until ref_transaction_commit() is called.\n\n2. The LMDB write lock is database-wide. This means that a single\n writer, even if modifying only a single reference, blocks all\n writes to all references.\n\nTherefore I am worried that the LMDB backend will be susceptable to more\nlock conflicts than the files backend.\n\nWould would be the disadvantage of letting ref_transaction_update() and\nits friends build up `struct ref_update` objects in memory (much like\nthey do in the files backend), and only start the LMDB write transaction\nwhen ref_transaction_commit() is called? Obviously it would cost some\nmore bookkeeping, but that code is already written. This would be much\ncloser to the files backend model.\n\nThis is where I stopped for the day.\n\nI have a bunch of small stylistic and formatting suggestions regarding\nthis patch in particular. Rather than describe them all in prose, I just\nmade the suggested changes and pushed them to my GitHub fork [1], branch\n\"refs-be-lmdb-housecleaning\". Feel free to squash the ones you want into\nthis commit.\n\nA lot of these problems were found by gcc, which I apparently have\nconfigured more strictly than yours. You might want to pick some options\nfrom my config.mak [2] to make gcc enforce some of the Git project\npolicies automatically. (Every dev on this list probably has his/her own\nvariation on this file.)\n\nI apologize again for not having reviewed your patches more promptly. I\nreally think it is good and important work. I've just been so busy with\nother things and I knew it would take a couple of solid days of work to\nget through it. I'll try to review the rest tomorrow, but since I don't\nknow anything about the LMDB API, the review will either be superficial\nor it will take a long time...\n\nLet me finish up by stating (also to remind myself!) that it is not\ncrucial to get everything perfect in the first iteration. As long as the\ndatabase format is correct, other things can be improved over time. This\nreference backend will inevitably go though a longish period when it is\nconsidered experimental, during which we will have time to hammer bugs\nout of it.\n\nMichael\n\n[1] https://github.com/mhagger/git\n[2] https://gist.github.com/mhagger/b321e423481b3de405d2\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271144","messageId":"xmqqh9m5z224.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"20151003173930.GA25904@spirit","subject":"Re: [PATCH v2 43/43] refs: tests for db backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-05T16:56:51Z","receivedAt":"2015-10-05T16:56:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dennis Kaarsemaker writes:\n\n> On Mon, Sep 28, 2015 at 06:02:18PM -0400, David Turner wrote:\n>> Add tests for the database backend.\n>> \n>> Signed-off-by: David Turner \n>> ---\n>> t/t1460-refs-be-db.sh | 1103 ++++++++++++++++++++++++++++++++++++++++++\n>> t/t1470-refs-be-db-reflog.sh | 353 ++++++++++++++\n>> 2 files changed, 1456 insertions(+)\n>> create mode 100755 t/t1460-refs-be-db.sh\n>> create mode 100755 t/t1470-refs-be-db-reflog.sh\n>\n> These break 'make test' on builds without the db backend. Maybe squash\n> in something like the following:\n>\n> diff --git a/t/t1460-refs-be-db.sh b/t/t1460-refs-be-db.sh\n> index f13b0f0..c8222ed 100755\n> --- a/t/t1460-refs-be-db.sh\n> +++ b/t/t1460-refs-be-db.sh\n> @@ -9,6 +9,11 @@ test_description='Test lmdb refs backend'\n> TEST_NO_CREATE_REPO=1\n> . ./test-lib.sh\n> \n> +if ! test -e ../../test-refs-be-lmdb; then\n> +\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n> +\ttest_done\n> +fi\n\nThe idea is sound, but $TRASH_DIRECTORY (i.e. $(cwd) there) is not\nnecessarily two subdirectories down from the build repository root\n(cf. --root= parameter to the test scripts), so you need to\naccount for that.\n\nThanks.\n"},{"id":"271146","messageId":"xmqq8u7hz0qw.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"56122EB6.2020501@alum.mit.edu","subject":"Re: [PATCH v2 18/43] refs-be-files.c: add a backend method structure with transaction functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-05T17:25:11Z","receivedAt":"2015-10-05T17:25:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n>> +/* refs backends */\n>> +typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n>\n> Hmmm, I thought our convention was to define typedefs for functions\n> themselves, not for the pointer-to-function; e.g.,\n>\n> typedef struct ref_transaction *ref_transaction_begin_fn(struct\n> strbuf *err);\n>\n> (which would require `struct ref_be` to be changed to\n>\n> ref_transaction_begin_fn *transaction_begin;\n>\n> etc.) But now as I grep through the code it looks like both conventions\n> are used. So never mind :-)\n\nWell spotted. My recollection is the same and we do prefer the\nlatter (I think all early function typedefs by Linus were done that\nway). It would be better to correct existing mistakes we added over\ntime and certainly not add more of them in new code.\n\nThanks.\n"},{"id":"271159","messageId":"1444076073.5158.10.camel@twopensource.com","threadId":"40445","inReplyTo":"56123279.4060605@alum.mit.edu","subject":"Re: [PATCH v2 22/43] refs-be-files.c: add do_for_each_per_worktree_ref","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-05T20:14:33Z","receivedAt":"2015-10-05T20:14:33Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 10:19 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:01 AM, David Turner wrote:\n> > Alternate refs backends might still use files to store per-worktree\n> > refs. So the files backend's ref-loading infrastructure should be\n> > available to those backends, just for use on per-worktree refs. Add\n> > do_for_each_per_worktree_ref, which iterates over per-worktree refs.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > refs-be-files.c | 15 ++++++++++++---\n> > refs.h | 11 +++++++++++\n> > 2 files changed, 23 insertions(+), 3 deletions(-)\n> > \n> > diff --git a/refs-be-files.c b/refs-be-files.c\n> > index eb18a20..dc89289 100644\n> > --- a/refs-be-files.c\n> > +++ b/refs-be-files.c\n> > @@ -587,9 +587,6 @@ static void sort_ref_dir(struct ref_dir *dir)\n> > \tdir->sorted = dir->nr = i;\n> > }\n> > \n> > -/* Include broken references in a do_for_each_ref*() iteration: */\n> > -#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n> > -\n> > /*\n> > * Return true iff the reference described by entry can be resolved to\n> > * an object in the database. Emit a warning if the referred-to\n> > [...]\n> > diff --git a/refs.h b/refs.h\n> > index 5875fe5..09d140d 100644\n> > --- a/refs.h\n> > +++ b/refs.h\n> > @@ -152,6 +152,12 @@ struct ref_transaction;\n> > */\n> > #define REF_BAD_NAME 0x08\n> > \n> > +/* Include broken references in a do_for_each_ref*() iteration */\n> > +#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n> \n> Why do you move this definition from refs-be-files.c?\n\nBecause the lmdb backend will need it.\n\n> > +\n> > +/* Only include per-worktree refs in a do_for_each_ref*() iteration */\n> > +#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n> \n> And why do you define this one here instead of in refs-be-files.c?\n\nBecause other backends might need it (even though at present lmdb\ndoesn't).\n"},{"id":"271161","messageId":"1444076584.5158.12.camel@twopensource.com","threadId":"40445","inReplyTo":"5611FCA4.9030809@alum.mit.edu","subject":"Re: [PATCH v2 01/43] refs.c: create a public version of verify_refname_available","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-05T20:23:04Z","receivedAt":"2015-10-05T20:23:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 06:29 +0200, Michael Haggerty wrote:\n> + * skip must be sorted.\n> > + */\n> \n> This comment is approximately a copy of the comment for\n> verify_refname_available_dir(). It seems unnecessary to keep both of\n> them (and is also a small maintenance burden). I suggest you shorten \n> the\n> comment at verify_refname_available_dir() and make it refer to the\n> comment for this function for the details.\n\nWill do, thanks.\n"},{"id":"271163","messageId":"1444076803.5158.13.camel@twopensource.com","threadId":"40445","inReplyTo":"5611FDD9.1050405@alum.mit.edu","subject":"Re: [PATCH v2 02/43] refs: make repack_without_refs and is_branch public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-05T20:26:43Z","receivedAt":"2015-10-05T20:26:43Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 06:34 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:01 AM, David Turner wrote:\n> > Signed-off-by: Ronnie Sahlberg \n> > Signed-off-by: David Turner \n> > ---\n> > refs.c | 9 +--------\n> > refs.h | 13 +++++++++++++\n> > 2 files changed, 14 insertions(+), 8 deletions(-)\n> > \n> > diff --git a/refs.c b/refs.c\n> > index ce551e9..2741cc5 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -2826,14 +2826,7 @@ int pack_refs(unsigned int flags)\n> > \treturn 0;\n> > }\n> > \n> > -/*\n> > - * Rewrite the packed-refs file, omitting any refs listed in\n> > - * 'refnames'. On error, leave packed-refs unchanged, write an error\n> > - * message to 'err', and return a nonzero value.\n> > - *\n> > - * The refs in 'refnames' needn't be sorted. `err` must not be NULL.\n> > - */\n> > -static int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n> > +int repack_without_refs(struct string_list *refnames, struct strbuf *err)\n> \n> I looked for the corresponding change to remove `static` from\n> is_branch(). Apparently that function already had external linkage, even\n> though it was not listed in the header file. As a convenience to readers\n> you might note that peculiarity in the commit message.\n\nWill fix, thnaks.\n"},{"id":"271172","messageId":"1444077424.5158.14.camel@twopensource.com","threadId":"40445","inReplyTo":"561236E7.1050508@alum.mit.edu","subject":"Re: [PATCH v2 28/43] refs.c: add ref backend init function","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-05T20:37:04Z","receivedAt":"2015-10-05T20:37:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 10:37 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > The file backend doesn't need this function, but other backends might.\n> > \n> > Signed-off-by: David Turner \n> > Signed-off-by: Ronnie Sahlberg \n> > ---\n> > refs-be-files.c | 1 +\n> > refs.c | 4 +++-\n> > refs.h | 4 +++-\n> > 3 files changed, 7 insertions(+), 2 deletions(-)\n> > \n> > diff --git a/refs-be-files.c b/refs-be-files.c\n> > index 37e244a..eaa74b6 100644\n> > --- a/refs-be-files.c\n> > +++ b/refs-be-files.c\n> > @@ -3737,6 +3737,7 @@ int reflog_expire(const char *refname, const unsigned char *sha1,\n> > struct ref_be refs_be_files = {\n> > \tNULL,\n> > \t\"files\",\n> > +\tNULL,\n> > \tfiles_transaction_begin,\n> > \tfiles_transaction_update,\n> > \tfiles_transaction_create,\n> > diff --git a/refs.c b/refs.c\n> > index 769574d..9ce10b7 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -19,13 +19,15 @@ struct ref_be *refs_backends = &refs_be_files;\n> > /*\n> > * This function is used to switch to an alternate backend.\n> > */\n> > -int set_refs_backend(const char *name)\n> > +int set_refs_backend(const char *name, void *init_data)\n> > {\n> > \tstruct ref_be *be;\n> > \n> > \tfor (be = refs_backends; be; be = be->next)\n> > \t\tif (!strcmp(be->name, name)) {\n> > \t\t\tthe_refs_backend = be;\n> > +\t\t\tif (be->init_backend)\n> > +\t\t\t\tbe->init_backend(init_data);\n> \n> I don't like that this virtual function, alone among all of them\n> introduced so far, is allowed to be NULL. That seems non-obvious and\n> something extra that devs have to remember.\n> \n> I think it would be better for the files backend to define a do-nothing\n> function that can be stuck in this slot.\n\nWill change, thanks.\n"},{"id":"271190","messageId":"1444090810.7739.3.camel@twopensource.com","threadId":"40445","inReplyTo":"xmqqh9m5z224.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 43/43] refs: tests for db backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T00:20:10Z","receivedAt":"2015-10-06T00:20:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 09:56 -0700, Junio C Hamano wrote:\n> Dennis Kaarsemaker writes:\n> \n> > On Mon, Sep 28, 2015 at 06:02:18PM -0400, David Turner wrote:\n> >> Add tests for the database backend.\n> >> \n> >> Signed-off-by: David Turner \n> >> ---\n> >> t/t1460-refs-be-db.sh | 1103 ++++++++++++++++++++++++++++++++++++++++++\n> >> t/t1470-refs-be-db-reflog.sh | 353 ++++++++++++++\n> >> 2 files changed, 1456 insertions(+)\n> >> create mode 100755 t/t1460-refs-be-db.sh\n> >> create mode 100755 t/t1470-refs-be-db-reflog.sh\n> >\n> > These break 'make test' on builds without the db backend. Maybe squash\n> > in something like the following:\n> >\n> > diff --git a/t/t1460-refs-be-db.sh b/t/t1460-refs-be-db.sh\n> > index f13b0f0..c8222ed 100755\n> > --- a/t/t1460-refs-be-db.sh\n> > +++ b/t/t1460-refs-be-db.sh\n> > @@ -9,6 +9,11 @@ test_description='Test lmdb refs backend'\n> > TEST_NO_CREATE_REPO=1\n> > . ./test-lib.sh\n> > \n> > +if ! test -e ../../test-refs-be-lmdb; then\n> > +\tskip_all=\"Skipping lmdb refs backend tests, lmdb backend not built\"\n> > +\ttest_done\n> > +fi\n> \n> The idea is sound, but $TRASH_DIRECTORY (i.e. $(cwd) there) is not\n> necessarily two subdirectories down from the build repository root\n> (cf. --root= parameter to the test scripts), so you need to\n> account for that.\n\nwill fix these, thanks Dennis and Junio.\n"},{"id":"271191","messageId":"1444090829.7739.4.camel@twopensource.com","threadId":"40445","inReplyTo":"xmqq8u7hz0qw.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 18/43] refs-be-files.c: add a backend method structure with transaction functions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T00:20:29Z","receivedAt":"2015-10-06T00:20:29Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 10:25 -0700, Junio C Hamano wrote:\n> Michael Haggerty writes:\n> \n> >> +/* refs backends */\n> >> +typedef struct ref_transaction *(*ref_transaction_begin_fn)(struct strbuf *err);\n> >\n> > Hmmm, I thought our convention was to define typedefs for functions\n> > themselves, not for the pointer-to-function; e.g.,\n> >\n> > typedef struct ref_transaction *ref_transaction_begin_fn(struct\n> > strbuf *err);\n> >\n> > (which would require `struct ref_be` to be changed to\n> >\n> > ref_transaction_begin_fn *transaction_begin;\n> >\n> > etc.) But now as I grep through the code it looks like both conventions\n> > are used. So never mind :-)\n> \n> Well spotted. My recollection is the same and we do prefer the\n> latter (I think all early function typedefs by Linus were done that\n> way). It would be better to correct existing mistakes we added over\n> time and certainly not add more of them in new code.\n\nWill fix, thanks.\n"},{"id":"271192","messageId":"1444091314.7739.5.camel@twopensource.com","threadId":"40445","inReplyTo":"56123DAD.9010204@alum.mit.edu","subject":"Re: [PATCH v2 39/43] refs: break out a ref conflict check","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T00:28:34Z","receivedAt":"2015-10-06T00:28:34Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 11:06 +0200, Michael Haggerty wrote:\n> > +\t/*\n> > +\t * Check for entries in extras that start with \"$refname/\". We\n> > +\t * do that by looking for the place where \"$refname/\" would be\n> > +\t * inserted in extras. If there is an entry at that position\n> > +\t * that starts with \"$refname/\" and is not in skip, then we\n> > +\t * have a conflict.\n> > +\t */\n> \n> Would you please turn this comment (or something like it) into a\n> docstring for this function in refs.h?\n\nYes, thanks.\n"},{"id":"271193","messageId":"1444094198.7739.15.camel@twopensource.com","threadId":"40445","inReplyTo":"5612653F.5070502@alum.mit.edu","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T01:16:38Z","receivedAt":"2015-10-06T01:16:38Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 13:55 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > Add a new option, --refs-backend-type, to allow the ref backend type to\n> > be set on new clones.\n> > \n> > Submodules must use the same ref backend as the parent repository, so\n> > we also pass the --refs-backend-type option option when cloning\n> > submodules.\n> > \n> > Signed-off-by: David Turner \n> > ---\n> > Documentation/git-clone.txt | 4 ++++\n> > builtin/clone.c | 27 +++++++++++++++++++++++++--\n> > builtin/submodule--helper.c | 5 ++++-\n> > cache.h | 1 +\n> > refs.c | 2 ++\n> > 5 files changed, 36 insertions(+), 3 deletions(-)\n> > \n> > [...]\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index 3e14491..d489a87 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > [...]\n> > @@ -744,6 +762,11 @@ static void write_config(struct string_list *config)\n> > \t\t\t\t\t write_one_config, NULL) < 0)\n> > \t\t\tdie(\"unable to write parameters to config file\");\n> > \t}\n> > +\n> > +\tif (refs_backend_type &&\n> > +\t write_one_config(\"core.refs-backend-type\",\n> > +\t\t\t refs_backend_type, NULL) < 0)\n> > +\t\t\tdie(\"unable to write backend parameter to config file\");\n> > }\n> \n> Our config variable names don't use separated-with-hyphens but rather\n> wordsruntogether (though in documentation this would be written\n> core.refsBackendType, because the actual names are case-insensitive).\n\nWill fix, thanks.\n"},{"id":"271194","messageId":"1444094644.7739.20.camel@twopensource.com","threadId":"40445","inReplyTo":"56123CE3.9070909@alum.mit.edu","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T01:24:04Z","receivedAt":"2015-10-06T01:24:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 11:03 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > Because HEAD and stash are per-worktree, other backends need to\n> > go through the files backend to manage these refs and their reflogs.\n> > \n> > To enable this, we make some files backend functions public.\n> \n> I have a bad feeling about this change.\n> \n> Naively I would expect a reference backend that cannot handle its own\n> (e.g.) stash to instantiate internally a files backend object and to\n> delegate stash-related calls to that object. That way neither class's\n> interface has to be changed.\n> \n> Here you are adding a separate interface to the files backend. That\n> seems like a more complicated and less flexible design. But I'm open to\n> be persuaded otherwise...\n\nOK, I've switched over most of these to use an internal files backend.\n\nWhile doing so, I thought I needed to keep files_log_ref_write, \nbecause we need to handle refs that cross over between the lmdb \nbackend and the files backend.\n\nBut on reflection, I have realized that I'm doing cross-backend refs \nwrong anyway (I'm prematurely committing the files backend ones, and\ndoing it without locks). So I think I need to rethink this a bit. \nProbably I can split out a files transaction using REF_NODEREF or \nsomething.\n\nWill do something about this.\n"},{"id":"271195","messageId":"1444094977.7739.24.camel@twopensource.com","threadId":"40445","inReplyTo":"5612439E.4080200@alum.mit.edu","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T01:29:37Z","receivedAt":"2015-10-06T01:29:37Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 11:32 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > Add a new option, --refs-backend-type, to allow the ref backend type to\n> > be set on new clones.\n> > \n> > Submodules must use the same ref backend as the parent repository, so\n> > we also pass the --refs-backend-type option option when cloning\n> > submodules.\n> \n> If I'm reading this correctly, you propose to add a new configuration\n> setting,\n> \n> core.refs-backend-type\n> \n> to record which reference backend the repository is using.\n> \n> (By the way, please document any new config settings in\n> Documentation/config.txt.)\n\nFixed, thanks.\n\n> This does not provide adequate backwards compatibility. Suppose I create\n> a repository with a DB backend, then later try to access the repository\n> using an older version of Git (or JGit or libgit2 or ...) that doesn't\n> know about this configuration setting. It will think it is a normal\n> repository that only has a couple of references (HEAD, stash, etc). If I\n> run `git gc` using the older version of git, *poof* all of the objects\n> referred to by other branches will be garbage-collected.\n> \n> Therefore, I don't think this can be merged without a bump to\n> core.repositoryformatversion. Such a bump will tell well-behaved older\n> Git clients keep their hands off the repository. (Of course repositories\n> that use the files backend can continue using\n> core.repositoryformatversion 0.)\n> \n> I thought Peff proposed a way to do such a bump, including a way to\n> extend repositories one by one with new features. But that was something\n> that we were chatting about off-list.\n> \n> I haven't reviewed the actual code in this patch yet but I wanted to get\n> the above comment on your radar.\n> \n> Michael\n\nI'll fix this to upgrade to v=1 when the lmdb refs backend is in use,\nand to give sensible error messages in a v1 repo if built without LMDB.\n"},{"id":"271196","messageId":"20151006015806.GA4972@sigill.intra.peff.net","threadId":"40445","inReplyTo":"1444094977.7739.24.camel@twopensource.com","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-10-06T01:58:07Z","receivedAt":"2015-10-06T01:58:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 05, 2015 at 09:29:37PM -0400, David Turner wrote:\n\n> > Therefore, I don't think this can be merged without a bump to\n> > core.repositoryformatversion. Such a bump will tell well-behaved older\n> > Git clients keep their hands off the repository. (Of course repositories\n> > that use the files backend can continue using\n> > core.repositoryformatversion 0.)\n> > \n> > I thought Peff proposed a way to do such a bump, including a way to\n> > extend repositories one by one with new features. But that was something\n> > that we were chatting about off-list.\n> > \n> > I haven't reviewed the actual code in this patch yet but I wanted to get\n> > the above comment on your radar.\n> > \n> > Michael\n> \n> I'll fix this to upgrade to v=1 when the lmdb refs backend is in use,\n> and to give sensible error messages in a v1 repo if built without LMDB.\n\nI think the relevant series is:\n\n http://article.gmane.org/gmane.comp.version-control.git/272447\n\nIt did not seem too controversial, but it mostly got dropped amidst the\nrelease, and I haven't reposted it yet.\n\n-Peff\n"},{"id":"271237","messageId":"CAO2U3QhdzLfx8qfnOHgXTWit5x7+TQNn0=xozouA4m-0+B1rtw@mail.gmail.com","threadId":"40445","inReplyTo":"1443477738-32023-35-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v2 34/43] refs.c: make struct ref_transaction generic","fromName":"Michael Blume","fromEmail":"blume.mike@gmail.com","sentAt":"2015-10-06T17:43:00Z","receivedAt":"2015-10-06T17:43:00Z","isPatch":true,"sender":{"key":"blume.mike@gmail.com","avatar":"https://gravatar.com/avatar/1a7b440e1d942425ff4098ac7fc15b86b30cecaa56e1692a7ef8b5939ba25ea7?d=mp&s=160"},"body":"This triggers a lot of build warnings on my mac, basically in every\nfile that uses refs.h:\n\n CC archive.o\nIn file included from archive.c:2:\n./refs.h:635:16: warning: redefinition of typedef\n'ref_transaction_free_fn' is a C11 feature [-Wtypedef-redefinition]\ntypedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n ^\n./refs.h:613:16: note: previous definition is here\ntypedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n ^\n1 warning generated.\n\nOn Mon, Sep 28, 2015 at 3:02 PM, David Turner wrote:\n> Alternate ref backends might need different data for transactions. Make\n> struct ref_transaction an empty struct, and let backends define their\n> own structs which extend it.\n>\n> Signed-off-by: David Turner \n> ---\n> refs-be-files.c | 24 +++++++++++++++++-------\n> refs.h | 8 ++++++--\n> 2 files changed, 23 insertions(+), 9 deletions(-)\n>\n> diff --git a/refs-be-files.c b/refs-be-files.c\n> index 0a76c8e..3f2d194 100644\n> --- a/refs-be-files.c\n> +++ b/refs-be-files.c\n> @@ -3218,7 +3218,8 @@ enum ref_transaction_state {\n> * consist of checks and updates to multiple references, carried out\n> * as atomically as possible. This structure is opaque to callers.\n> */\n> -struct ref_transaction {\n> +struct files_ref_transaction {\n> + struct ref_transaction base;\n> struct ref_update **updates;\n> size_t alloc;\n> size_t nr;\n> @@ -3229,13 +3230,16 @@ static struct ref_transaction *files_transaction_begin(struct strbuf *err)\n> {\n> assert(err);\n>\n> - return xcalloc(1, sizeof(struct ref_transaction));\n> + return xcalloc(1, sizeof(struct files_ref_transaction));\n> }\n>\n> -static void files_transaction_free(struct ref_transaction *transaction)\n> +static void files_transaction_free(struct ref_transaction *trans)\n> {\n> int i;\n>\n> + struct files_ref_transaction *transaction =\n> + (struct files_ref_transaction *)trans;\n> +\n> if (!transaction)\n> return;\n>\n> @@ -3247,7 +3251,7 @@ static void files_transaction_free(struct ref_transaction *transaction)\n> free(transaction);\n> }\n>\n> -static struct ref_update *add_update(struct ref_transaction *transaction,\n> +static struct ref_update *add_update(struct files_ref_transaction *transaction,\n> const char *refname)\n> {\n> size_t len = strlen(refname) + 1;\n> @@ -3259,7 +3263,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,\n> return update;\n> }\n>\n> -static int files_transaction_update(struct ref_transaction *transaction,\n> +static int files_transaction_update(struct ref_transaction *trans,\n> const char *refname,\n> const unsigned char *new_sha1,\n> const unsigned char *old_sha1,\n> @@ -3267,6 +3271,8 @@ static int files_transaction_update(struct ref_transaction *transaction,\n> struct strbuf *err)\n> {\n> struct ref_update *update;\n> + struct files_ref_transaction *transaction =\n> + (struct files_ref_transaction *)trans;\n>\n> assert(err);\n>\n> @@ -3350,10 +3356,12 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n> return 0;\n> }\n>\n> -static int files_transaction_commit(struct ref_transaction *transaction,\n> +static int files_transaction_commit(struct ref_transaction *trans,\n> struct strbuf *err)\n> {\n> int ret = 0, i;\n> + struct files_ref_transaction *transaction =\n> + (struct files_ref_transaction *)trans;\n> int n = transaction->nr;\n> struct ref_update **updates = transaction->updates;\n> struct string_list refs_to_delete = STRING_LIST_INIT_NODUP;\n> @@ -3517,10 +3525,12 @@ static int ref_present(const char *refname,\n> return string_list_has_string(affected_refnames, refname);\n> }\n>\n> -static int files_initial_transaction_commit(struct ref_transaction *transaction,\n> +static int files_initial_transaction_commit(struct ref_transaction *trans,\n> struct strbuf *err)\n> {\n> int ret = 0, i;\n> + struct files_ref_transaction *transaction =\n> + (struct files_ref_transaction *)trans;\n> int n = transaction->nr;\n> struct ref_update **updates = transaction->updates;\n> struct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> diff --git a/refs.h b/refs.h\n> index 64dba64..02001ef 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -130,7 +130,7 @@ extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);\n> *\n> * Calling sequence\n> * ----------------\n> - * - Allocate and initialize a `struct ref_transaction` by calling\n> + * - Allocate and initialize a transaction by calling\n> * `ref_transaction_begin()`.\n> *\n> * - List intended ref updates by calling functions like\n> @@ -156,7 +156,10 @@ extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);\n> * The message is appended to err without first clearing err.\n> * err will not be '\\n' terminated.\n> */\n> -struct ref_transaction;\n> +\n> +struct ref_transaction {\n> + /* ref backends should extend this */\n> +};\n>\n> /*\n> * Bit values set in the flags argument passed to each_ref_fn():\n> @@ -629,6 +632,7 @@ typedef int (*reflog_expire_fn)(const char *refname, const unsigned char *sha1,\n> void *policy_cb_data);\n>\n> /* resolution functions */\n> +typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n> typedef const char *(*resolve_ref_unsafe_fn)(const char *ref,\n> int resolve_flags,\n> unsigned char *sha1, int *flags);\n> --\n> 2.4.2.644.g97b850b-twtrsrc\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"271238","messageId":"1444154006.7739.28.camel@twopensource.com","threadId":"40445","inReplyTo":"CAO2U3QhdzLfx8qfnOHgXTWit5x7+TQNn0=xozouA4m-0+B1rtw@mail.gmail.com","subject":"Re: [PATCH v2 34/43] refs.c: make struct ref_transaction generic","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T17:53:26Z","receivedAt":"2015-10-06T17:53:26Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2015-10-06 at 10:43 -0700, Michael Blume wrote:\n> This triggers a lot of build warnings on my mac, basically in every\n> file that uses refs.h:\n> \n> CC archive.o\n> In file included from archive.c:2:\n> ./refs.h:635:16: warning: redefinition of typedef\n> 'ref_transaction_free_fn' is a C11 feature [-Wtypedef-redefinition]\n> typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n> ^\n> ./refs.h:613:16: note: previous definition is here\n> typedef void (*ref_transaction_free_fn)(struct ref_transaction *transaction);\n> ^\n> 1 warning generated.\n\n\nThanks. I'll fix that when I reroll.\n"},{"id":"271239","messageId":"1444154963.7739.30.camel@twopensource.com","threadId":"40445","inReplyTo":"20151006015806.GA4972@sigill.intra.peff.net","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-06T18:09:23Z","receivedAt":"2015-10-06T18:09:23Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 21:58 -0400, Jeff King wrote:\n> On Mon, Oct 05, 2015 at 09:29:37PM -0400, David Turner wrote:\n> \n> > > Therefore, I don't think this can be merged without a bump to\n> > > core.repositoryformatversion. Such a bump will tell well-behaved older\n> > > Git clients keep their hands off the repository. (Of course repositories\n> > > that use the files backend can continue using\n> > > core.repositoryformatversion 0.)\n> > > \n> > > I thought Peff proposed a way to do such a bump, including a way to\n> > > extend repositories one by one with new features. But that was something\n> > > that we were chatting about off-list.\n> > > \n> > > I haven't reviewed the actual code in this patch yet but I wanted to get\n> > > the above comment on your radar.\n> > > \n> > > Michael\n> > \n> > I'll fix this to upgrade to v=1 when the lmdb refs backend is in use,\n> > and to give sensible error messages in a v1 repo if built without LMDB.\n> \n> I think the relevant series is:\n> \n> http://article.gmane.org/gmane.comp.version-control.git/272447\n> \n> It did not seem too controversial, but it mostly got dropped amidst the\n> release, and I haven't reposted it yet.\n\nThat patch will work perfectly for this use case. I'll add it to my\nseries when I reroll, and set an extension.\n"},{"id":"271251","messageId":"1444181145.7739.70.camel@twopensource.com","threadId":"40445","inReplyTo":"56123CE3.9070909@alum.mit.edu","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-07T01:25:45Z","receivedAt":"2015-10-07T01:25:45Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 11:03 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > Because HEAD and stash are per-worktree, other backends need to\n> > go through the files backend to manage these refs and their reflogs.\n> > \n> > To enable this, we make some files backend functions public.\n> \n> I have a bad feeling about this change.\n> \n> Naively I would expect a reference backend that cannot handle its own\n> (e.g.) stash to instantiate internally a files backend object and to\n> delegate stash-related calls to that object. That way neither class's\n> interface has to be changed.\n> \n> Here you are adding a separate interface to the files backend. That\n> seems like a more complicated and less flexible design. But I'm open to\n> be persuaded otherwise...\n\nAfter some thought, here's a summary of the problem:\n\nSome writes are cross-backend writes. For example, if HEAD is symref to\nrefs/head/master, a commit is a cross-backend write (HEAD itself is not\nupdated, but its reflog is). Ronnie's design of the ref backend\nstructure did not account for cross-backend writes, because we didn't\nhave per-worktree refs at the time (there was only HEAD, and there was\nonly one copy of it).\n\nCross-backend writes are complicated because there is no way to tell a\nbackend to do only part of a ref update -- for instance, to tell the\nfiles backend to update HEAD and HEAD's reflog but not\nrefs/heads/master. Maybe we could set a flag that would do this, but\nthe synchronization would be fairly complicated. For instance, an\nupdate to HEAD might need to confirm the old sha for HEAD, meaning that\nwe couldn't do the db write first. But if we move the db write second,\nthen when the db code goes to do its check of the HEAD sha, it might see\na new value. Perhaps there's a way to make it work, but it seems\nfragile/complex.\n\nRight now, for cross-backend reads/writes, the lmdb code cheats. It\nsimply does the write directly and immediately. This means that these\nportions of transactions cannot be rolled back. That's clearly bad. \n\n\n\nThe simplest solution would be for the lmdb code to simply acquire\nlocks, and write to lock files, and then commit those lock files just\nbefore the db transaction commits. Then the lmdb code would handle all\nof the orchestration without the files backend having to be rewritten to\nhandle this case.\n\nWe would still like to use files_log_ref_write, since that handles the\nformatting details of writing to file-based reflogs. If you want, we\ncould break that (and its callees) out to separate refs-be-common.[ch]\nfiles, as you suggested for some #defines. (The calls to\nfiles_log_ref_write would also have to move to transaction_commit as\nwell, but I think that's straightforward).\n\nDoes this seem reasonable to you?\n"},{"id":"271252","messageId":"1444182660.7739.77.camel@twopensource.com","threadId":"40445","inReplyTo":"56129B77.1030409@alum.mit.edu","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-07T01:51:00Z","receivedAt":"2015-10-07T01:51:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-10-05 at 17:47 +0200, Michael Haggerty wrote:\n> On 09/29/2015 12:02 AM, David Turner wrote:\n> > Add a database backend for refs using LMDB. This backend runs git\n> > for-each-ref about 30% faster than the files backend with fully-packed\n> > refs on a repo with ~120k refs. It's also about 4x faster than using\n> > fully-unpacked refs. In addition, and perhaps more importantly , it\n> > avoids case-conflict issues on OS X.\n> > \n> > LMDB has a few features that make it suitable for usage in git:\n> > \n> > 1. It is relatively lightweight; it requires only one header file, and\n> > the library code takes under 64k at runtime.\n> > \n> > 2. It is well-tested: it's been used in OpenLDAP for years.\n> > \n> > 3. It's very fast. LMDB's benchmarks show that it is among\n> > the fastest key-value stores.\n> > \n> > 4. It has a relatively simple concurrency story; readers don't\n> > block writers and writers don't block readers.\n> \n> It would be great if you would go into more detail about this point in\n> the permanent technical documentation\n> (Documentation/technical/refs-be-lmdb.txt): how exactly do readers &\n> writers interact with each others, and how do writers interact with\n> other writers?\n\nI'll add a bit of text on that.\n\n> I think you have said before that if one writer holds the write lock on\n> the DB, then other writers fail immediately. Is that correct? If so, is\n> that something that can be adjusted? I think it would be preferable for\n> the second writer to retry acquiring the write lock for a little while\n> with a timeout (as we now do when trying to acquire the packed-refs\n> lock). Otherwise you could have the unhappy situation that somebody\n> spends a long time pushing a packfile to a server, only to have the\n> reference update be rejected at the last moment due to a lock conflict\n> with another process that was touching completely different references.\n> We already do before/after consistency checks when updating references,\n> so you wouldn't even have to add such code in the backend yourself.\n\nNo, the second writer waits for the first writer to unlock (or for it to\ncrash).\n\n> > +Keys for refs are the name of the ref (e.g. refs/heads/master).\n> > +Values are the value of the ref\n> > +(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).\n> \n> Please document more unambiguously whether the value is stored as\n> 40-char hexadecimal or 20-byte binary.\n\nWill fix.\n\n> Do you store \"peeled\" reference values for tags, as is done in\n> packed-refs? I think that is an important optimization.\n\nNo. Do you happen to know in what situations this is a performance\nbenefit, so that I can benchmark? I suspect it would matter much less\nfor the LMDB backend, because lookups are pretty quick.\n\n> > +All per-worktree refs (refs/bisect/* and HEAD) are store using\n> > +the traditional files-based backend.\n> > +\n> > +Reflogs are stored as a series of database entries.\n> > +\n> > +For an empty reflog, there is a \"header\" entry to show that a reflog\n> > +exists.\n> \n> What key is used for the \"header\" entry? Does it have a value?\n\nWill fix.\n\n> > + For non-empty reflogs, there is one entry per logged ref\n> > +update. The key format is logs/[refname]\\0[timestamp]. The timestamp\n> > +is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.\n> > +This means that reflog entries are chronologically ordered. Because\n> > +LMDB is a btree database, we can efficiently iterate over these keys.\n> \n> Currently we discard the reflog for a reference when the reference is\n> deleted. This is mostly due to a limitation of the filesystem-based\n> storage scheme--otherwise the reflog left over from a deleted reference\n> could prevent the creation of a reflog for another reference whose name\n> overlaps with it (in the sense of is_refname_available()).\n> \n> It is a pretty nasty limitation because it is one of the few ways that\n> your actions can cause Git to lose data in a way that it is hard to\n> recover from.\n> \n> Have you thought about removing this limitation in the lbdb backend?\n\nI'm going for feature parity first. We can always add new functionality\nlater. This particular function would be pretty straightforward to add,\nI think.\n\n> > +Reflog values are in the same format as the original files-based\n> > +reflog.\n> > +\n> > +Weaknesses:\n> > +-----------\n> > +\n> > +The reflog format is somewhat inefficient: a binary format could store\n> > +reflog date/time information in somewhat less space.\n> > +\n> > +The rsync and file:// transports don't work yet, because they\n> > +don't use the refs API.\n> \n> Do they fail gracefully?\n\nNot particularly gracefully.\n\nrsync: link_stat \"/home/dturner/git/t/trash\ndirectory.t5510-fetch/.git/packed-refs\" failed: No such file or\ndirectory (2)\nrsync error: some files/attrs were not transferred (see previous errors)\n(code 23) at main.c(1183) [sender=3.1.1]\nfatal: Could not run rsync to get refs\n-------------\n\nThe problem is that rsync on the client assumes that packed-refs exists.\nWe could hack it to also check for refdb.\n\n> > +#ifdef USE_LIBLMDB\n> > +\t\t\trefs_backend_type = refdb_data.refs_backend_type;\n> > +\t\t\tregister_refs_backend(&refs_be_lmdb);\n> > +\t\t\tset_refs_backend(refs_backend_type, &refdb_data);\n> > +#else\n> > +\t\t\tdie(\"Git was not built with USE_LIBLMDB, so the db refs backend is not available\");\n> \n> s/db/lmdb/\n> \n> I'm somewhat surprised that you only register the lmdb backend if it is\n> used in the main repo. I would expect the backend to be registered\n> unconditionally on startup. The cost is trivial, isn't it?\n\nYeah, but this was the easiest place to do it.\n\n> It appears that you recognize \"lmdb\" but you treat a missing value *or\n> any other value* like \"files\". I think you should be more conservative\n> here and die() if you see a backend type that you don't recognize.\n\nWill fix, thanks.\n\n> > diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\n> > index 888c34a..9761a1a 100755\n> > --- a/contrib/workdir/git-new-workdir\n> > +++ b/contrib/workdir/git-new-workdir\n> > @@ -28,6 +28,8 @@ git_dir=$(cd \"$orig_git\" 2>/dev/null &&\n> > git rev-parse --git-dir 2>/dev/null) ||\n> > die \"Not a git repository: \\\"$orig_git\\\"\"\n> > \n> > +test -d $git_dir/refdb && die \"git-new-workdir is incompatible with the refs db backend\"\n> > +\n> \n> It would be safer to check $(git config core.refs-backend-type) to avoid\n> depending on an implementation detail of the LMDB backend (and as a bit\n> of safety against any backends that might be added in the future).\n\nWill fix, thanks.\n\n> > +static void init_env(MDB_env **env, const char *path)\n> > +{\n> > +\tint ret;\n> > +\tif (*env)\n> > +\t\treturn;\n> > +\n> > +\tif ((ret = mdb_env_create(env)) != MDB_SUCCESS)\n> > +\t\tdie(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n> \n> The LMDB docs seem to say that their functions \"return a non-zero error\n> value on failure and 0 on success\". So can we avoid writing MDB_SUCCESS\n> everywhere and just compare to zero?\n\nSure.\n\n> Also, the Git project CodingGuidelines say to avoid assignments in the\n> condition of an \"if\" statement.\n> \n> So all in all I think these should be written like\n> \n> ret = mdb_env_create(env);\n> if (ret)\n> die(\"mdb_env_create failed: %s\", mdb_strerror(ret));\n\nOK.\n\n> > +\n> > +int ref_update_cmp(const void *entry, const void *entry_or_key, const void *keydata)\n> \n> The usual convention for a \"cmp\" function is that it returns <0, =0, or\n> >0 depending on the relative ordering of its arguments. This one can\n> only be used for equality/inequality. Please add a comment warning the\n> reader of this (and maybe rename it, e.g., to ref_update_not_equal(), to\n> further reduce the chance of confusion).\n\nWill fix.\n\n> > +\tif (total_commands_run != last_commands_run) {\n> > +\t\t/*\n> > +\t\t * Since each transaction sees a consistent view of\n> > +\t\t * the db, downstream processes that write the db\n> > +\t\t * won't be seen in this transaction. We don't know\n> > +\t\t * whether any given downstream process has made any\n> > +\t\t * writes, so if there have been any downstream processes,\n> > +\t\t * we had better reopen the transaction.\n> > +\t\t */\n> \n> Is it possible to tell from the database whether there has been a write\n> since the current read transaction was started? \n\nActually, yes. Fixed.\n\n> It seems like this would\n> be a more reliable method, given that an unrelated process might have\n> updated references while this command was running, or a long-running\n> background process (like Duy's background `git gc --auto`) might still\n> continue running even after the original command exits.\n> \n> Of course in the end checks like this can never 100% prevent races. The\n> only way, it would seem, would be to check-and-set during a single write\n> transaction that holds a write lock on the DB.\n> \n> That's exactly what we do for file-based references: ideally, the only\n> allowed changes are compare-and-set operations that are made atomic by\n> holding a lock while it is occurring (one or more loose reference\n> lockfiles and/or the packed-refs lockfile). That is why our ref_updates\n> have old_sha1 and new_sha1, and old_sha1 is always used for operations\n> that start before a lock is held.\n> \n> So I guess my question is this: to what extent are you relying on this\n> total_commands_run mechanism for correctness, vs. just using it to avoid\n> using reference values that are *really* old?\n\nSome tests fail without it, unfortunately. (I can't remember which ones\nnow).\n\n> It seems to me that a very important feature of Git is the following:\n> \n> Two processes *must not* block each other (except possibly\n> for a very brief time) if they are updating disjoint sets\n> of references.\n>\n> With the file-backed reference backend, this feature is accomplished via\n> the following mechanisms:\n> \n> * Locks are only held for very short periods of time, and never when\n> calling other processes.\n> \n> * Locks usually only apply to single references (loose references).\n> Sometimes the packed-refs file has to be locked, but in such cases\n> a second writer retries the lock acquisition for long enough that\n> it should usually succeed.\n> \n> * If a reference value was checked before a lock was held, then it\n> is always re-checked after acquiring the lock to make sure that\n> another process didn't change it in the meantime. (If the\n> reference *was* changed from its expected value, it is ok to\n> die().) This is necessary to preserve safety even if a lock is\n> not held for the duration of a command.\n> \n> In particular, locks are only held for the duration of the call to\n> ref_transaction_commit(), *not* while building up a transaction using\n> ref_transaction_update() etc.\n> \n> I am afraid that the LMDB backend as currently implemented will be more\n> subject to locking conflicts because the LMDB locks are broader in both\n> time and space:\n> \n> 1. You are holding the LMDB transaction open for longer (during the\n> whole time that the ref_transaction is being built up). This is\n> worse simply because it is a longer time. But it is also worse\n> because the process that is building up the transaction might be\n> doing other things while holding the write lock. Have you checked\n> whether any callers do things that could change references (for\n> example, invoke user hooks) between the calls to\n> ref_transaction_begin() and ref_transaction_commit()? With the\n> files backend this isn't an issue because the lock isn't acquired\n> until ref_transaction_commit() is called.\n> \n> 2. The LMDB write lock is database-wide. This means that a single\n> writer, even if modifying only a single reference, blocks all\n> writes to all references.\n> \n> Therefore I am worried that the LMDB backend will be susceptable to more\n> lock conflicts than the files backend.\n> \n> [What] would be the disadvantage of letting ref_transaction_update() and\n> its friends build up `struct ref_update` objects in memory (much like\n> they do in the files backend), and only start the LMDB write transaction\n> when ref_transaction_commit() is called? Obviously it would cost some\n> more bookkeeping, but that code is already written. This would be much\n> closer to the files backend model.\n> \n> This is where I stopped for the day.\n\nI think you have convinced me to switch over to this method. Unless\nsomething weird comes up while I'm doing the switch, in which case I'll\nsend email.\n\n> I have a bunch of small stylistic and formatting suggestions regarding\n> this patch in particular. Rather than describe them all in prose, I just\n> made the suggested changes and pushed them to my GitHub fork [1], branch\n> \"refs-be-lmdb-housecleaning\". Feel free to squash the ones you want into\n> this commit.\n\nI took them all. THanks.\n\n> A lot of these problems were found by gcc, which I apparently have\n> configured more strictly than yours. You might want to pick some options\n> from my config.mak [2] to make gcc enforce some of the Git project\n> policies automatically. (Every dev on this list probably has his/her own\n> variation on this file.)\n\nI've adopted yours, thanks.\n\n> I apologize again for not having reviewed your patches more promptly. I\n> really think it is good and important work. I've just been so busy with\n> other things and I knew it would take a couple of solid days of work to\n> get through it. I'll try to review the rest tomorrow, but since I don't\n> know anything about the LMDB API, the review will either be superficial\n> or it will take a long time...\n\nThanks for the hard work here. I know this is a big chunk of code to\nreview, and I'm grateful for the thoughts.\n"},{"id":"271258","messageId":"56154194.9050607@alum.mit.edu","threadId":"40445","inReplyTo":"1444181145.7739.70.camel@twopensource.com","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-07T16:00:20Z","receivedAt":"2015-10-07T16:00:20Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 10/07/2015 03:25 AM, David Turner wrote:\n> On Mon, 2015-10-05 at 11:03 +0200, Michael Haggerty wrote:\n>> On 09/29/2015 12:02 AM, David Turner wrote:\n>>> Because HEAD and stash are per-worktree, other backends need to\n>>> go through the files backend to manage these refs and their reflogs.\n>>>\n>>> To enable this, we make some files backend functions public.\n>>\n>> I have a bad feeling about this change.\n>>\n>> Naively I would expect a reference backend that cannot handle its own\n>> (e.g.) stash to instantiate internally a files backend object and to\n>> delegate stash-related calls to that object. That way neither class's\n>> interface has to be changed.\n>>\n>> Here you are adding a separate interface to the files backend. That\n>> seems like a more complicated and less flexible design. But I'm open to\n>> be persuaded otherwise...\n> \n> After some thought, here's a summary of the problem:\n> \n> Some writes are cross-backend writes. For example, if HEAD is symref to\n> refs/head/master, a commit is a cross-backend write (HEAD itself is not\n> updated, but its reflog is). Ronnie's design of the ref backend\n> structure did not account for cross-backend writes, because we didn't\n> have per-worktree refs at the time (there was only HEAD, and there was\n> only one copy of it).\n> \n> Cross-backend writes are complicated because there is no way to tell a\n> backend to do only part of a ref update -- for instance, to tell the\n> files backend to update HEAD and HEAD's reflog but not\n> refs/heads/master. Maybe we could set a flag that would do this, but\n> the synchronization would be fairly complicated. For instance, an\n> update to HEAD might need to confirm the old sha for HEAD, meaning that\n> we couldn't do the db write first. But if we move the db write second,\n> then when the db code goes to do its check of the HEAD sha, it might see\n> a new value. Perhaps there's a way to make it work, but it seems\n> fragile/complex.\n> \n> Right now, for cross-backend reads/writes, the lmdb code cheats. It\n> simply does the write directly and immediately. This means that these\n> portions of transactions cannot be rolled back. That's clearly bad. \n\nThat's a really good point.\n\nI hate to break it to you, but the handling of symrefs in Git is already\na mess. HEAD is the only symref that I would really trust to work\ncorrectly all the time. So I think that changes needn't be judged on\nwhether they handle symrefs perfectly. They should just not break them\nin any dramatic new ways.\n\nSo, you pointed out the problem that HEAD (a per-worktree reference) can\nbe a symref that points at a shared reference. In fact, I think when\nHEAD is symbolic it is only allowed to point at a branch under\nrefs/heads, so this particular problem is pretty well-constrained.\n\nAre there other cases of cross-backend writes? I suppose there could be\na symref elsewhere among the per-worktree references that points at a\nshared reference. But I can't think of any cases where this is done by\nstandard Git. Not that it is forbidden; I just don't think it is done by\nany of the standard tools.\n\nOr there could be a symref among the shared references that points at a\nper-worktree reference. But AFAIK the only other symrefs that are in\ncommon use are the refs/remotes/*/HEAD symrefs, and they always point at\nreferences within the same (shared) namespace.\n\nIf everything that I've said is correct, then my opinion is that it\nwould be perfectly adequate if your code would handle the specific case\nof HEAD (by hook or by crook), and if there are any other cross-backend\nsymrefs, just die with a message stating that such usage is unsupported.\nJunio, do you think that would be acceptable?\n\n> The simplest solution would be for the lmdb code to simply acquire\n> locks, and write to lock files, and then commit those lock files just\n> before the db transaction commits. Then the lmdb code would handle all\n> of the orchestration without the files backend having to be rewritten to\n> handle this case.\n\nWouldn't that essentially be re-implementing the files backend? I must\nbe missing something.\n\n> [...]\n\nBTW I just realized that if one backend should delegate to another, then\nthe primary backend should be the per-worktree backend and it should\ndelegate to the common backend. I think I described things the other way\naround in my earlier message. This makes more sense because it is\nacceptable for per-worktree references to refer to common references but\nnot vice versa.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271264","messageId":"xmqqk2qyvbm0.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"56154194.9050607@alum.mit.edu","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-07T17:20:55Z","receivedAt":"2015-10-07T17:20:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n>> Right now, for cross-backend reads/writes, the lmdb code cheats. It\n>> simply does the write directly and immediately. This means that these\n>> portions of transactions cannot be rolled back. That's clearly bad. \n>\n> That's a really good point.\n>\n> I hate to break it to you, but the handling of symrefs in Git is already\n> a mess. HEAD is the only symref that I would really trust to work\n> correctly all the time. So I think that changes needn't be judged on\n> whether they handle symrefs perfectly. They should just not break them\n> in any dramatic new ways.\n> ...\n> If everything that I've said is correct, then my opinion is that it\n> would be perfectly adequate if your code would handle the specific case\n> of HEAD (by hook or by crook), and if there are any other cross-backend\n> symrefs, just die with a message stating that such usage is unsupported.\n> Junio, do you think that would be acceptable?\n\nI find it a very sensible approach.\n\n> BTW I just realized that if one backend should delegate to another, then\n> the primary backend should be the per-worktree backend and it should\n> delegate to the common backend. I think I described things the other way\n> around in my earlier message. This makes more sense because it is\n> acceptable for per-worktree references to refer to common references but\n> not vice versa.\n\nSounds good.\n\nThanks.\n"},{"id":"271270","messageId":"561564EC.8070704@alum.mit.edu","threadId":"40445","inReplyTo":"1444182660.7739.77.camel@twopensource.com","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-07T18:31:08Z","receivedAt":"2015-10-07T18:31:08Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 10/07/2015 03:51 AM, David Turner wrote:\n> On Mon, 2015-10-05 at 17:47 +0200, Michael Haggerty wrote:\n>> On 09/29/2015 12:02 AM, David Turner wrote:\n>>> Add a database backend for refs using LMDB. [...]\n>>\n>> I think you have said before that if one writer holds the write lock on\n>> the DB, then other writers fail immediately. Is that correct? If so, is\n>> that something that can be adjusted? I think it would be preferable for\n>> the second writer to retry acquiring the write lock for a little while\n>> with a timeout (as we now do when trying to acquire the packed-refs\n>> lock). Otherwise you could have the unhappy situation that somebody\n>> spends a long time pushing a packfile to a server, only to have the\n>> reference update be rejected at the last moment due to a lock conflict\n>> with another process that was touching completely different references.\n>> We already do before/after consistency checks when updating references,\n>> so you wouldn't even have to add such code in the backend yourself.\n> \n> No, the second writer waits for the first writer to unlock (or for it to\n> crash).\n\nCool, that's better behavior.\n\n> [...]\n>> Do you store \"peeled\" reference values for tags, as is done in\n>> packed-refs? I think that is an important optimization.\n> \n> No. Do you happen to know in what situations this is a performance\n> benefit, so that I can benchmark? I suspect it would matter much less\n> for the LMDB backend, because lookups are pretty quick.\n\nThe reference lookup speed is not relevant here. \"Peeling\" is applied to\nreferences that point at tag objects (a.k.a. annotated tags). It means\nthat the tag object is looked up to see what *it* points at (recursively\nif necessary) and the result is stored to the packed-refs file in a\nspecially-formatted extra line that looks like\n\n 17f9f635c101aef03874e1de1d8d0322187494b3 refs/tags/v2.6.0\n ^be08dee9738eaaa0423885ed189c2b6ad8368cf0\n\nI think a good command to benchmark would be `git show-refs -d` in a\nrepository with a number of annotated tags. This command's output is\nsimilar to the output of `git ls-remote ` and also comes up\nduring reference negotiation when fetching (so its performance is\ndefinitely not moot).\n\n> [...]\n>> Currently we discard the reflog for a reference when the reference is\n>> deleted. [...]\n>> Have you thought about removing this limitation in the lbdb backend?\n> \n> I'm going for feature parity first. We can always add new functionality\n> later. This particular function would be pretty straightforward to add,\n> I think.\n\n+1\n\n> [...]\n>>> +The rsync and file:// transports don't work yet, because they\n>>> +don't use the refs API.\n>>\n>> Do they fail gracefully?\n> \n> Not particularly gracefully.\n> \n> rsync: link_stat \"/home/dturner/git/t/trash\n> directory.t5510-fetch/.git/packed-refs\" failed: No such file or\n> directory (2)\n> rsync error: some files/attrs were not transferred (see previous errors)\n> (code 23) at main.c(1183) [sender=3.1.1]\n> fatal: Could not run rsync to get refs\n> -------------\n> \n> The problem is that rsync on the client assumes that packed-refs exists.\n> We could hack it to also check for refdb.\n\nI guess this is something that will have to be improved sooner or later,\nthough I guess not as a precondition for merging this patch series.\n\n> [...]\n>> I'm somewhat surprised that you only register the lmdb backend if it is\n>> used in the main repo. I would expect the backend to be registered\n>> unconditionally on startup. The cost is trivial, isn't it?\n> \n> Yeah, but this was the easiest place to do it.\n\nOK.\n\n> [...]\n\nI'm really happy about your work.\n\nRegarding strategy: I think a good approach would be to get as much of\nthe preparatory work as possible (the abstraction and separation of\nrefs-be-files) to the point where it can be merged before there is too\nmuch more code churn in the area. That work is not very controversial, I\nthink, and letting it wait for a long time will increase the likelihood\nof conflicts with other people's changes. The refs-be-lmdb patches, on\nthe other hand, (1) will take longer to get polished, (2) will take\nlonger to review because other people are not familiar with LDMB, and\n(3) won't bitrot very fast anyway because they don't overlap as much\nwith areas that other people are likely to work on. So I would advocate\nworking on those at a more deliberate pace and planning for them to be\nmerged as a separate batch.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271271","messageId":"xmqqvbaits2i.fsf@gitster.mtv.corp.google.com","threadId":"40445","inReplyTo":"561564EC.8070704@alum.mit.edu","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-07T19:08:21Z","receivedAt":"2015-10-07T19:08:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> I'm really happy about your work.\n>\n> Regarding strategy: I think a good approach would be to get as much of\n> the preparatory work as possible (the abstraction and separation of\n> refs-be-files) to the point where it can be merged before there is too\n> much more code churn in the area. That work is not very controversial, I\n> think, and letting it wait for a long time will increase the likelihood\n> of conflicts with other people's changes. The refs-be-lmdb patches, on\n> the other hand, (1) will take longer to get polished, (2) will take\n> longer to review because other people are not familiar with LDMB, and\n> (3) won't bitrot very fast anyway because they don't overlap as much\n> with areas that other people are likely to work on. So I would advocate\n> working on those at a more deliberate pace and planning for them to be\n> merged as a separate batch.\n\nI agree with you on all counts.\n\nMy wish is for you to play an interim maintainer starting around\nweek #7 (Nov 9th) of this cycle and merge the early \"preparatory\"\npart to 'next' before I come back around the end of November ;-)\nto be shipped as part of the release at the end of this cycle.\n\nThanks.\n"},{"id":"271272","messageId":"1444245629.8836.12.camel@twopensource.com","threadId":"40445","inReplyTo":"561564EC.8070704@alum.mit.edu","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-07T19:20:29Z","receivedAt":"2015-10-07T19:20:29Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-10-07 at 20:31 +0200, Michael Haggerty wrote:\n> On 10/07/2015 03:51 AM, David Turner wrote:\n> > On Mon, 2015-10-05 at 17:47 +0200, Michael Haggerty wrote:\n> >> On 09/29/2015 12:02 AM, David Turner wrote:\n> >>> Add a database backend for refs using LMDB. [...]\n> >>\n> >> I think you have said before that if one writer holds the write lock on\n> >> the DB, then other writers fail immediately. Is that correct? If so, is\n> >> that something that can be adjusted? I think it would be preferable for\n> >> the second writer to retry acquiring the write lock for a little while\n> >> with a timeout (as we now do when trying to acquire the packed-refs\n> >> lock). Otherwise you could have the unhappy situation that somebody\n> >> spends a long time pushing a packfile to a server, only to have the\n> >> reference update be rejected at the last moment due to a lock conflict\n> >> with another process that was touching completely different references.\n> >> We already do before/after consistency checks when updating references,\n> >> so you wouldn't even have to add such code in the backend yourself.\n> > \n> > No, the second writer waits for the first writer to unlock (or for it to\n> > crash).\n> \n> Cool, that's better behavior.\n> \n> > [...]\n> >> Do you store \"peeled\" reference values for tags, as is done in\n> >> packed-refs? I think that is an important optimization.\n> > \n> > No. Do you happen to know in what situations this is a performance\n> > benefit, so that I can benchmark? I suspect it would matter much less\n> > for the LMDB backend, because lookups are pretty quick.\n> \n> The reference lookup speed is not relevant here. \"Peeling\" is applied to\n> references that point at tag objects (a.k.a. annotated tags). It means\n> that the tag object is looked up to see what *it* points at (recursively\n> if necessary) and the result is stored to the packed-refs file in a\n> specially-formatted extra line that looks like\n> \n> 17f9f635c101aef03874e1de1d8d0322187494b3 refs/tags/v2.6.0\n> ^be08dee9738eaaa0423885ed189c2b6ad8368cf0\n> \n> I think a good command to benchmark would be `git show-refs -d` in a\n> repository with a number of annotated tags. This command's output is\n> similar to the output of `git ls-remote ` and also comes up\n> during reference negotiation when fetching (so its performance is\n> definitely not moot).\n> \n> > [...]\n> >> Currently we discard the reflog for a reference when the reference is\n> >> deleted. [...]\n> >> Have you thought about removing this limitation in the lbdb backend?\n> > \n> > I'm going for feature parity first. We can always add new functionality\n> > later. This particular function would be pretty straightforward to add,\n> > I think.\n> \n> +1\n> \n> > [...]\n> >>> +The rsync and file:// transports don't work yet, because they\n> >>> +don't use the refs API.\n> >>\n> >> Do they fail gracefully?\n> > \n> > Not particularly gracefully.\n> > \n> > rsync: link_stat \"/home/dturner/git/t/trash\n> > directory.t5510-fetch/.git/packed-refs\" failed: No such file or\n> > directory (2)\n> > rsync error: some files/attrs were not transferred (see previous errors)\n> > (code 23) at main.c(1183) [sender=3.1.1]\n> > fatal: Could not run rsync to get refs\n> > -------------\n> > \n> > The problem is that rsync on the client assumes that packed-refs exists.\n> > We could hack it to also check for refdb.\n> \n> I guess this is something that will have to be improved sooner or later,\n> though I guess not as a precondition for merging this patch series.\n> \n> > [...]\n> >> I'm somewhat surprised that you only register the lmdb backend if it is\n> >> used in the main repo. I would expect the backend to be registered\n> >> unconditionally on startup. The cost is trivial, isn't it?\n> > \n> > Yeah, but this was the easiest place to do it.\n> \n> OK.\n> \n> > [...]\n> \n> I'm really happy about your work.\n> \n> Regarding strategy: I think a good approach would be to get as much of\n> the preparatory work as possible (the abstraction and separation of\n> refs-be-files) to the point where it can be merged before there is too\n> much more code churn in the area. That work is not very controversial, I\n> think, and letting it wait for a long time will increase the likelihood\n> of conflicts with other people's changes. The refs-be-lmdb patches, on\n> the other hand, (1) will take longer to get polished, (2) will take\n> longer to review because other people are not familiar with LDMB, and\n> (3) won't bitrot very fast anyway because they don't overlap as much\n> with areas that other people are likely to work on. So I would advocate\n> working on those at a more deliberate pace and planning for them to be\n> merged as a separate batch.\n\nWorks for me. \n\nWould you like me to start sending those as a separate series, or shall\nI keep it as one series and let you split it as you choose?\n"},{"id":"271275","messageId":"1444251311.8836.22.camel@twopensource.com","threadId":"40445","inReplyTo":"56154194.9050607@alum.mit.edu","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-07T20:55:11Z","receivedAt":"2015-10-07T20:55:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-10-07 at 18:00 +0200, Michael Haggerty wrote:\n> On 10/07/2015 03:25 AM, David Turner wrote:\n> > On Mon, 2015-10-05 at 11:03 +0200, Michael Haggerty wrote:\n> >> On 09/29/2015 12:02 AM, David Turner wrote:\n> >>> Because HEAD and stash are per-worktree, other backends need to\n> >>> go through the files backend to manage these refs and their reflogs.\n> >>>\n> >>> To enable this, we make some files backend functions public.\n> >>\n> >> I have a bad feeling about this change.\n> >>\n> >> Naively I would expect a reference backend that cannot handle its own\n> >> (e.g.) stash to instantiate internally a files backend object and to\n> >> delegate stash-related calls to that object. That way neither class's\n> >> interface has to be changed.\n> >>\n> >> Here you are adding a separate interface to the files backend. That\n> >> seems like a more complicated and less flexible design. But I'm open to\n> >> be persuaded otherwise...\n> > \n> > After some thought, here's a summary of the problem:\n> > \n> > Some writes are cross-backend writes. For example, if HEAD is symref to\n> > refs/head/master, a commit is a cross-backend write (HEAD itself is not\n> > updated, but its reflog is). Ronnie's design of the ref backend\n> > structure did not account for cross-backend writes, because we didn't\n> > have per-worktree refs at the time (there was only HEAD, and there was\n> > only one copy of it).\n> > \n> > Cross-backend writes are complicated because there is no way to tell a\n> > backend to do only part of a ref update -- for instance, to tell the\n> > files backend to update HEAD and HEAD's reflog but not\n> > refs/heads/master. Maybe we could set a flag that would do this, but\n> > the synchronization would be fairly complicated. For instance, an\n> > update to HEAD might need to confirm the old sha for HEAD, meaning that\n> > we couldn't do the db write first. But if we move the db write second,\n> > then when the db code goes to do its check of the HEAD sha, it might see\n> > a new value. Perhaps there's a way to make it work, but it seems\n> > fragile/complex.\n> > \n> > Right now, for cross-backend reads/writes, the lmdb code cheats. It\n> > simply does the write directly and immediately. This means that these\n> > portions of transactions cannot be rolled back. That's clearly bad. \n> \n> That's a really good point.\n> \n> I hate to break it to you, but the handling of symrefs in Git is already\n> a mess. HEAD is the only symref that I would really trust to work\n> correctly all the time. So I think that changes needn't be judged on\n> whether they handle symrefs perfectly. They should just not break them\n> in any dramatic new ways.\n> \n> So, you pointed out the problem that HEAD (a per-worktree reference) can\n> be a symref that points at a shared reference. In fact, I think when\n> HEAD is symbolic it is only allowed to point at a branch under\n> refs/heads, so this particular problem is pretty well-constrained.\n> \n> Are there other cases of cross-backend writes? I suppose there could be\n> a symref elsewhere among the per-worktree references that points at a\n> shared reference. But I can't think of any cases where this is done by\n> standard Git. Not that it is forbidden; I just don't think it is done by\n> any of the standard tools.\n\nAnother case would be an update-ref command that updates both\nrefs/bisect/something and refs/heads/something. \n\nI don't think git ever does this by default, but anyone can issue a\nweird update-ref command if they feel like it.\n\n> Or there could be a symref among the shared references that points at a\n> per-worktree reference. But AFAIK the only other symrefs that are in\n> common use are the refs/remotes/*/HEAD symrefs, and they always point at\n> references within the same (shared) namespace.\n> \n> If everything that I've said is correct, then my opinion is that it\n> would be perfectly adequate if your code would handle the specific case\n> of HEAD (by hook or by crook), and if there are any other cross-backend\n> symrefs, just die with a message stating that such usage is unsupported.\n> Junio, do you think that would be acceptable?\n\nHm. I don't think it's significantly easier to handle just HEAD than\nit would be to handle all cases. But I'll see what happens as I write\nthe code.\n\n> > The simplest solution would be for the lmdb code to simply acquire\n> > locks, and write to lock files, and then commit those lock files just\n> > before the db transaction commits. Then the lmdb code would handle all\n> > of the orchestration without the files backend having to be rewritten to\n> > handle this case.\n> \n> Wouldn't that essentially be re-implementing the files backend? I must\n> be missing something.\n\nThere would be some amount of reimplementation, yes. But if we assume\nthat the number of per-worktree refs is relatively small, we could make\nsome simplification. But actually, see below.\n\n> > [...]\n> \n> BTW I just realized that if one backend should delegate to another, then\n> the primary backend should be the per-worktree backend and it should\n> delegate to the common backend. I think I described things the other way\n> around in my earlier message. This makes more sense because it is\n> acceptable for per-worktree references to refer to common references but\n> not vice versa.\n\nI think I might have a good way to deal with this:\n\nIf we're going to switch the lmdb transaction code over to accumulate\nupdates and then do them as one batch, then probably all other\nbackends will work the same way. So maybe there is no need for all of\nthese backend functions:\n\n\tref_transaction_begin_fn *transaction_begin;\n\tref_transaction_update_fn *transaction_update;\n\tref_transaction_create_fn *transaction_create;\n\tref_transaction_delete_fn *transaction_delete;\n\tref_transaction_verify_fn *transaction_verify;\n\nInstead, the generic refs code will accumulate updates in a struct\nref_update. Instead of a lock, the ref_update struct will have a void\npointer that backends can use for per-update data (such as the lock).\nThe generic code can also handle rejecting duplicate ref updates.\n\nThe per-backend transaction_commit method will just take a struct\nref_transaction (that is, what the current patchset calls a\nfiles_ref_transaction) -- basically, a list of ref_updates -- and\nattempt to apply it.\n\nWhile we're doing this, the generic ref code can detect an update to\nHEAD, and replace it with an update to whatever HEAD points to (if HEAD\nis a symref). Then it can call files_log_ref_write to write to HEAD's\nreflog, if the main transaction commits successfully. If HEAD is not a\nsymref, the generic code can just move the HEAD update over to the files\nbackend.\n\nDoes this make sense?\n"},{"id":"271284","messageId":"561598F0.8050004@alum.mit.edu","threadId":"40445","inReplyTo":"1444245629.8836.12.camel@twopensource.com","subject":"Re: [PATCH v2 42/43] refs: add LMDB refs backend","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-07T22:13:04Z","receivedAt":"2015-10-07T22:13:04Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 10/07/2015 09:20 PM, David Turner wrote:\n> On Wed, 2015-10-07 at 20:31 +0200, Michael Haggerty wrote:\n>> [...]\n>> I'm really happy about your work.\n>>\n>> Regarding strategy: I think a good approach would be to get as much of\n>> the preparatory work as possible (the abstraction and separation of\n>> refs-be-files) to the point where it can be merged before there is too\n>> much more code churn in the area. That work is not very controversial, I\n>> think, and letting it wait for a long time will increase the likelihood\n>> of conflicts with other people's changes. The refs-be-lmdb patches, on\n>> the other hand, (1) will take longer to get polished, (2) will take\n>> longer to review because other people are not familiar with LDMB, and\n>> (3) won't bitrot very fast anyway because they don't overlap as much\n>> with areas that other people are likely to work on. So I would advocate\n>> working on those at a more deliberate pace and planning for them to be\n>> merged as a separate batch.\n> \n> Works for me. \n> \n> Would you like me to start sending those as a separate series, or shall\n> I keep it as one series and let you split it as you choose?\n\nThat's really up to you, as the convenience tradeoff is mostly on your\nside. If you keep it as one series it is a tad easier for everybody to\nsee the whole idea as a continuous story. But it means that whenever you\nrewrite any commit in the series, you have to propagate the change\nthrough all of the commits, every time. Whereas if you break them up,\nyou have the option of letting the later patches idle for a while then\nto rebase them onto revision N of the earlier patch series in one big bang.\n\nBTW I didn't have the impression that the series has to be broken into\nmore than two or three subseries. Splitting off refs-be-files.{c,h} from\nrefs.{c,h} and creating the virtual function table is pretty much one\nunit of work and I see no sense splitting it artificially into separate\nparts. Adding refs-be-lmdb.{c,h} is only a couple of (big!) patches\nwhich, again, don't really need to be split any further. If you decide\nto implement the delegation thing for handling the split between\nper-worktree vs. shared references (even when both use the files\nbackend), that might be a third patch series between the other two.\n\nThe best dividing line to choose is between \"uncontroversial\" and\n\"possibly controversial\", because the former are more likely to succeed\nin a fast track.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271286","messageId":"5615A0F7.9090401@alum.mit.edu","threadId":"40445","inReplyTo":"1444251311.8836.22.camel@twopensource.com","subject":"Re: [PATCH v2 38/43] refs: make some files backend functions public","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-10-07T22:47:19Z","receivedAt":"2015-10-07T22:47:19Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 10/07/2015 10:55 PM, David Turner wrote:\n> On Wed, 2015-10-07 at 18:00 +0200, Michael Haggerty wrote:\n>>\n>> That's a really good point.\n>>\n>> I hate to break it to you, but the handling of symrefs in Git is already\n>> a mess. HEAD is the only symref that I would really trust to work\n>> correctly all the time. So I think that changes needn't be judged on\n>> whether they handle symrefs perfectly. They should just not break them\n>> in any dramatic new ways.\n>>\n>> So, you pointed out the problem that HEAD (a per-worktree reference) can\n>> be a symref that points at a shared reference. In fact, I think when\n>> HEAD is symbolic it is only allowed to point at a branch under\n>> refs/heads, so this particular problem is pretty well-constrained.\n>>\n>> Are there other cases of cross-backend writes? I suppose there could be\n>> a symref elsewhere among the per-worktree references that points at a\n>> shared reference. But I can't think of any cases where this is done by\n>> standard Git. Not that it is forbidden; I just don't think it is done by\n>> any of the standard tools.\n> \n> Another case would be an update-ref command that updates both\n> refs/bisect/something and refs/heads/something. \n> \n> I don't think git ever does this by default, but anyone can issue a\n> weird update-ref command if they feel like it.\n\nOh I was mostly worried about symbolic refs reaching across the divide.\nIf a transaction includes *non-symbolic* refs from both sides I think\nall you have to do is sort them into two piles and do one transaction\nfor each pile. You would probably have to sacrifice atomicity across the\ndividing line, but it's a bit unfair to expect atomic updates that span\ntwo possibly completely different backends.\n\nOK, you don't know for 100% sure that a reference is a symref before you\nhave locked it. But we've agreed that cross-repo symref support doesn't\nhave to be perfect (except for HEAD).\n\n>> Or there could be a symref among the shared references that points at a\n>> per-worktree reference. But AFAIK the only other symrefs that are in\n>> common use are the refs/remotes/*/HEAD symrefs, and they always point at\n>> references within the same (shared) namespace.\n>>\n>> If everything that I've said is correct, then my opinion is that it\n>> would be perfectly adequate if your code would handle the specific case\n>> of HEAD (by hook or by crook), and if there are any other cross-backend\n>> symrefs, just die with a message stating that such usage is unsupported.\n>> Junio, do you think that would be acceptable?\n> \n> Hm. I don't think it's significantly easier to handle just HEAD than\n> it would be to handle all cases. But I'll see what happens as I write\n> the code.\n\nI think the main simplification is having license not to worry about\nshared symrefs that could theoretically point at per-worktree\nreferences. Though I guess that's nonsensical anyway, so maybe it was\nalready obvious that you don't have to handle it.\n\n>>> The simplest solution would be for the lmdb code to simply acquire\n>>> locks, and write to lock files, and then commit those lock files just\n>>> before the db transaction commits. Then the lmdb code would handle all\n>>> of the orchestration without the files backend having to be rewritten to\n>>> handle this case.\n>>\n>> Wouldn't that essentially be re-implementing the files backend? I must\n>> be missing something.\n> \n> There would be some amount of reimplementation, yes. But if we assume\n> that the number of per-worktree refs is relatively small, we could make\n> some simplification. But actually, see below.\n> \n>>> [...]\n>>\n>> BTW I just realized that if one backend should delegate to another, then\n>> the primary backend should be the per-worktree backend and it should\n>> delegate to the common backend. I think I described things the other way\n>> around in my earlier message. This makes more sense because it is\n>> acceptable for per-worktree references to refer to common references but\n>> not vice versa.\n> \n> I think I might have a good way to deal with this:\n> \n> If we're going to switch the lmdb transaction code over to accumulate\n> updates and then do them as one batch, then probably all other\n> backends will work the same way. So maybe there is no need for all of\n> these backend functions:\n> \n> \tref_transaction_begin_fn *transaction_begin;\n> \tref_transaction_update_fn *transaction_update;\n> \tref_transaction_create_fn *transaction_create;\n> \tref_transaction_delete_fn *transaction_delete;\n> \tref_transaction_verify_fn *transaction_verify;\n> \n> Instead, the generic refs code will accumulate updates in a struct\n> ref_update. Instead of a lock, the ref_update struct will have a void\n> pointer that backends can use for per-update data (such as the lock).\n> The generic code can also handle rejecting duplicate ref updates.\n> \n> The per-backend transaction_commit method will just take a struct\n> ref_transaction (that is, what the current patchset calls a\n> files_ref_transaction) -- basically, a list of ref_updates -- and\n> attempt to apply it.\n> \n> While we're doing this, the generic ref code can detect an update to\n> HEAD, and replace it with an update to whatever HEAD points to (if HEAD\n> is a symref). Then it can call files_log_ref_write to write to HEAD's\n> reflog, if the main transaction commits successfully. If HEAD is not a\n> symref, the generic code can just move the HEAD update over to the files\n> backend.\n> \n> Does this make sense?\n\nThat makes a lot of sense. I like it.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"271441","messageId":"1444640408.41811.7.camel@dwim.me","threadId":"40445","inReplyTo":"1444154963.7739.30.camel@twopensource.com","subject":"Re: [PATCH v2 40/43] refs: allow ref backend to be set for clone","fromName":"Carlos Martín Nieto","fromEmail":"cmn@dwim.me","sentAt":"2015-10-12T09:00:08Z","receivedAt":"2015-10-12T09:00:08Z","isPatch":true,"sender":{"key":"cmn@dwim.me","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Tue, 2015-10-06 at 14:09 -0400, David Turner wrote:\n> On Mon, 2015-10-05 at 21:58 -0400, Jeff King wrote:\n> > On Mon, Oct 05, 2015 at 09:29:37PM -0400, David Turner wrote:\n> > \n> > > > Therefore, I don't think this can be merged without a bump to\n> > > > core.repositoryformatversion. Such a bump will tell well-\n> > > > behaved older\n> > > > Git clients keep their hands off the repository. (Of course\n> > > > repositories\n> > > > that use the files backend can continue using\n> > > > core.repositoryformatversion 0.)\n> > > > \n> > > > I thought Peff proposed a way to do such a bump, including a\n> > > > way to\n> > > > extend repositories one by one with new features. But that was\n> > > > something\n> > > > that we were chatting about off-list.\n> > > > \n> > > > I haven't reviewed the actual code in this patch yet but I\n> > > > wanted to get\n> > > > the above comment on your radar.\n> > > > \n> > > > Michael\n> > > \n> > > I'll fix this to upgrade to v=1 when the lmdb refs backend is in\n> > > use,\n> > > and to give sensible error messages in a v1 repo if built without\n> > > LMDB.\n> > \n> > I think the relevant series is:\n> > \n> >   http://article.gmane.org/gmane.comp.version-control.git/272447\n> > \n> > It did not seem too controversial, but it mostly got dropped amidst\n> > the\n> > release, and I haven't reposted it yet.\n> \n> That patch will work perfectly for this use case.  I'll add it to my\n> series when I reroll, and set an extension.\n\nThis is something I'm working on right now for libgit2 as well; not\nlmdb specifically but allowing user-provided backends, which would\nallow built-in ones as well.\n\nDid we ever decide on the format for these extensions? The series\nmentioned above has a couple of examples, but doesn't have any testing\nfor backend stuff. Do we have a concrete proposal for this? I was going\nto go for something like\n\nextensions.refbackend = \"lmdb\"\nextensions.odbbackend = \"psql\"\n\nand have backends register themselves by the \"lmdb\", \"psql\" or whatever\nformat, but if we have already decided something else which I missed,\nI'd swap over to that.\n\nCheers,\n   cmn\n"}]}