{"thread":{"id":"60240","subject":"[PATCH 0/3] Simple reftable backend","startedAt":"2023-09-18T17:59:46Z","lastAt":"2023-09-21T17:10:59Z","messageCount":15,"participants":["Han-Wen Nienhuys via GitGitGadget","Junio C Hamano","Han-Wen Nienhuys","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"481967","messageId":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":null,"subject":"[PATCH 0/3] Simple reftable backend","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-18T17:59:35Z","receivedAt":"2023-09-18T17:59:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This series comes from a conversation with Patrick Steinhardt at Gitlab, who\nhave an interest in a more scalable ref storage system.\n\nI unfortunately don't have business reasons to spend much time on this\nproject anymore, and the original, sweeping migration has too many open\nquestions.\n\nI thought of an alternate approach, and I wanted to show it as input to\ndiscussions at the contributor summit.\n\nI think the first part (\"refs: push lock management into packed backend\") is\na good improvement overall, and could be landed separately without much\ntweaking.\n\nThe second part (\"refs: alternate reftable ref backend implementation\") is\nsomething open for discussion: the alternative \"packed storage\" is based on\nreftable, but it could conceivably be a different type of database/file\nformat too. I think it's attractive to use reftable here, because over time\nmore data (symrefs, reflog) could be moved into reftable.\n\nHan-Wen Nienhuys (3):\n refs: push lock management into packed backend\n refs: move is_packed_transaction_needed out of packed-backend.c\n refs: alternate reftable ref backend implementation\n\n Makefile | 1 +\n config.mak.uname | 2 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs.c | 8 +\n refs/debug.c | 15 +\n refs/files-backend.c | 182 +++-\n refs/packed-backend.c | 150 +--\n refs/packed-backend.h | 19 -\n refs/refs-internal.h | 6 +\n refs/reftable-backend.c | 1745 +++++++++++++++++++++++++++++++\n refs/reftable-backend.h | 8 +\n 11 files changed, 1941 insertions(+), 197 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 refs/reftable-backend.h\n\n\nbase-commit: bda494f4043963b9ec9a1ecd4b19b7d1cd9a0518\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1574%2Fhanwen%2Fsimple-reftable-backend-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1574/hanwen/simple-reftable-backend-v1\nPull-Request: https://github.com/git/git/pull/1574\n-- \ngitgitgadget\n"},{"id":"481968","messageId":"dea0fbb139a82fe449a7edab6a8f445ce763d0c0.1695059978.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","subject":"[PATCH 1/3] refs: push lock management into packed backend","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-18T17:59:36Z","receivedAt":"2023-09-18T17:59:47Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nIntroduces a ref backend method transaction_begin, which for the\npacked backend takes packed-refs.lock.\n\nThis decouples the files backend from the packed backend, allowing the\nlatter to be swapped out by another ref backend.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs.c | 8 +++++\n refs/debug.c | 15 +++++++++\n refs/files-backend.c | 78 +++++++++++++++----------------------------\n refs/packed-backend.c | 41 ++++++++++++++---------\n refs/packed-backend.h | 10 ------\n refs/refs-internal.h | 5 +++\n 6 files changed, 79 insertions(+), 78 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex fcae5dddc60..a2d1012520e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1130,10 +1130,18 @@ struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,\n \t\t\t\t\t\t struct strbuf *err)\n {\n \tstruct ref_transaction *tr;\n+\tint ret = 0;\n \tassert(err);\n \n \tCALLOC_ARRAY(tr, 1);\n \ttr->ref_store = refs;\n+\n+\tif (refs->be->transaction_begin)\n+\t\tret = refs->be->transaction_begin(refs, tr, err);\n+\tif (ret) {\n+\t\tfree(tr);\n+\t\treturn NULL;\n+\t}\n \treturn tr;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex b7ffc4ce67e..964806e9301 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -41,6 +41,20 @@ static int debug_init_db(struct ref_store *refs, struct strbuf *err)\n \treturn res;\n }\n \n+static int debug_transaction_begin(struct ref_store *refs,\n+\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n+\tint res;\n+\ttransaction->ref_store = drefs->refs;\n+\tres = drefs->refs->be->transaction_begin(drefs->refs, transaction,\n+\t\t\t\t\t\t err);\n+\ttrace_printf_key(&trace_refs, \"transaction_begin: %d \\\"%s\\\"\\n\", res,\n+\t\t\t err->buf);\n+\treturn res;\n+}\n+\n static int debug_transaction_prepare(struct ref_store *refs,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n@@ -451,6 +465,7 @@ struct ref_storage_be refs_be_debug = {\n \t * has a function we should also have a wrapper for it here.\n \t * Test the output with \"GIT_TRACE_REFS=1\".\n \t */\n+\t.transaction_begin = debug_transaction_begin,\n \t.transaction_prepare = debug_transaction_prepare,\n \t.transaction_finish = debug_transaction_finish,\n \t.transaction_abort = debug_transaction_abort,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 341354182bb..4a6781af783 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1219,10 +1219,10 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tstruct ref_transaction *transaction;\n \n \ttransaction = ref_store_transaction_begin(refs->packed_ref_store, &err);\n-\tif (!transaction)\n+\tif (!transaction) {\n+\t\tdie(\"could not start transaction: %s\", err.buf);\n \t\treturn -1;\n-\n-\tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n+\t}\n \n \titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n \t\t\t\t\tthe_repository, 0);\n@@ -1262,8 +1262,6 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tref_transaction_free(transaction);\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \tprune_refs(refs, &refs_to_prune);\n \tstrbuf_release(&err);\n \treturn 0;\n@@ -1280,16 +1278,10 @@ static int files_delete_refs(struct ref_store *ref_store, const char *msg,\n \tif (!refnames->nr)\n \t\treturn 0;\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, &err))\n-\t\tgoto error;\n-\n \tif (refs_delete_refs(refs->packed_ref_store, msg, refnames, flags)) {\n-\t\tpacked_refs_unlock(refs->packed_ref_store);\n \t\tgoto error;\n \t}\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \tfor (i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \n@@ -2640,7 +2632,7 @@ out:\n \n struct files_transaction_backend_data {\n \tstruct ref_transaction *packed_transaction;\n-\tint packed_refs_locked;\n+\tint packed_transaction_needed;\n };\n \n /*\n@@ -2672,10 +2664,8 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \t\t\tstrbuf_release(&err);\n \t\t}\n \n-\t\tif (backend_data->packed_refs_locked)\n-\t\t\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \t\tfree(backend_data);\n+\t\ttransaction->backend_data = NULL;\n \t}\n \n \ttransaction->state = REF_TRANSACTION_CLOSED;\n@@ -2804,14 +2794,9 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t}\n \n \tif (packed_transaction) {\n-\t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n-\t\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t\tbackend_data->packed_refs_locked = 1;\n-\n-\t\tif (is_packed_transaction_needed(refs->packed_ref_store,\n-\t\t\t\t\t\t packed_transaction)) {\n+\t\tbackend_data->packed_transaction_needed = is_packed_transaction_needed(refs->packed_ref_store,\n+\t\t\t\t\t\t\t\t\t\t packed_transaction);\n+\t\tif (backend_data->packed_transaction_needed) {\n \t\t\tret = ref_transaction_prepare(packed_transaction, err);\n \t\t\t/*\n \t\t\t * A failure during the prepare step will abort\n@@ -2823,22 +2808,6 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\tref_transaction_free(packed_transaction);\n \t\t\t\tbackend_data->packed_transaction = NULL;\n \t\t\t}\n-\t\t} else {\n-\t\t\t/*\n-\t\t\t * We can skip rewriting the `packed-refs`\n-\t\t\t * file. But we do need to leave it locked, so\n-\t\t\t * that somebody else doesn't pack a reference\n-\t\t\t * that we are trying to delete.\n-\t\t\t *\n-\t\t\t * We need to disconnect our transaction from\n-\t\t\t * backend_data, since the abort (whether successful or\n-\t\t\t * not) will free it.\n-\t\t\t */\n-\t\t\tbackend_data->packed_transaction = NULL;\n-\t\t\tif (ref_transaction_abort(packed_transaction, err)) {\n-\t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n \t\t}\n \t}\n \n@@ -2940,13 +2909,24 @@ static int files_transaction_finish(struct ref_store *ref_store,\n \t * First delete any packed versions of the references, while\n \t * retaining the packed-refs lock:\n \t */\n-\tif (packed_transaction) {\n-\t\tret = ref_transaction_commit(packed_transaction, err);\n-\t\tref_transaction_free(packed_transaction);\n-\t\tpacked_transaction = NULL;\n-\t\tbackend_data->packed_transaction = NULL;\n-\t\tif (ret)\n-\t\t\tgoto cleanup;\n+\tif (backend_data->packed_transaction) { \n+\t\tif (backend_data->packed_transaction_needed) {\n+\t\t\tret = ref_transaction_commit(packed_transaction, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto cleanup;\n+\t\t\t/* TODO: leaks on error path. */\n+\t\t\tref_transaction_free(packed_transaction);\n+\t\t\tpacked_transaction = NULL;\n+\t\t\tbackend_data->packed_transaction = NULL;\n+\t\t} else {\n+\t\t\tret = ref_transaction_abort(packed_transaction, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto cleanup;\n+\n+\t\t\t/* transaction_commit doesn't free the data, but transaction_abort does. Go figure. */\n+\t\t\tpacked_transaction = NULL;\n+\t\t\tbackend_data->packed_transaction = NULL;\n+\t\t}\n \t}\n \n \t/* Now delete the loose versions of the references: */\n@@ -3087,16 +3067,10 @@ static int files_initial_transaction_commit(struct ref_store *ref_store,\n \t\t\t\t\t NULL);\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n-\n \tif (initial_ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t}\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n cleanup:\n \tif (packed_transaction)\n \t\tref_transaction_free(packed_transaction);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 59c78d7941f..5df7fa8004f 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1552,8 +1552,6 @@ int is_packed_transaction_needed(struct ref_store *ref_store,\n \n struct packed_transaction_backend_data {\n \t/* True iff the transaction owns the packed-refs lock. */\n-\tint own_lock;\n-\n \tstruct string_list updates;\n };\n \n@@ -1568,9 +1566,8 @@ static void packed_transaction_cleanup(struct packed_ref_store *refs,\n \t\tif (is_tempfile_active(refs->tempfile))\n \t\t\tdelete_tempfile(&refs->tempfile);\n \n-\t\tif (data->own_lock && is_lock_file_locked(&refs->lock)) {\n+\t\tif (is_lock_file_locked(&refs->lock)) {\n \t\t\tpacked_refs_unlock(&refs->base);\n-\t\t\tdata->own_lock = 0;\n \t\t}\n \n \t\tfree(data);\n@@ -1580,6 +1577,28 @@ static void packed_transaction_cleanup(struct packed_ref_store *refs,\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n }\n \n+static int packed_transaction_begin(struct ref_store *ref_store,\n+\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct packed_ref_store *refs = packed_downcast(\n+\t\t\tref_store,\n+\t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n+\t\t\t\"ref_transaction_begin\");\n+\tstruct packed_transaction_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstring_list_init_nodup(&data->updates);\n+\n+\tif (!is_lock_file_locked(&refs->lock)) {\n+\t\tif (packed_refs_lock(ref_store, 0, err))\n+\t\t\t/* todo: leaking data */\n+\t\t\treturn -1;\n+\t}\n+\ttransaction->backend_data = data;\n+\treturn 0;\n+}\n+\n static int packed_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n@@ -1588,7 +1607,7 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \t\t\tref_store,\n \t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n \t\t\t\"ref_transaction_prepare\");\n-\tstruct packed_transaction_backend_data *data;\n+\tstruct packed_transaction_backend_data *data = transaction->backend_data;\n \tsize_t i;\n \tint ret = TRANSACTION_GENERIC_ERROR;\n \n@@ -1601,11 +1620,6 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \t * do so itself.\n \t */\n \n-\tCALLOC_ARRAY(data, 1);\n-\tstring_list_init_nodup(&data->updates);\n-\n-\ttransaction->backend_data = data;\n-\n \t/*\n \t * Stick the updates in a string list by refname so that we\n \t * can sort them:\n@@ -1623,12 +1637,6 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \tif (ref_update_reject_duplicates(&data->updates, err))\n \t\tgoto failure;\n \n-\tif (!is_lock_file_locked(&refs->lock)) {\n-\t\tif (packed_refs_lock(ref_store, 0, err))\n-\t\t\tgoto failure;\n-\t\tdata->own_lock = 1;\n-\t}\n-\n \tif (write_with_updates(refs, &data->updates, err))\n \t\tgoto failure;\n \n@@ -1758,6 +1766,7 @@ struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_create,\n \t.init_db = packed_init_db,\n+\t.transaction_begin = packed_transaction_begin,\n \t.transaction_prepare = packed_transaction_prepare,\n \t.transaction_finish = packed_transaction_finish,\n \t.transaction_abort = packed_transaction_abort,\ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex 9dd8a344c34..ade3c8a5ac4 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -17,16 +17,6 @@ struct ref_store *packed_ref_store_create(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags);\n \n-/*\n- * Lock the packed-refs file for writing. Flags is passed to\n- * hold_lock_file_for_update(). Return 0 on success. On errors, write\n- * an error message to `err` and return a nonzero value.\n- */\n-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err);\n-\n-void packed_refs_unlock(struct ref_store *ref_store);\n-int packed_refs_is_locked(struct ref_store *ref_store);\n-\n /*\n * Return true if `transaction` really needs to be carried out against\n * the specified packed_ref_store, or false if it can be skipped\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 9db8aec4da8..87f8e8b51d6 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -531,6 +531,10 @@ typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n \n typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n \n+typedef int ref_transaction_begin_fn(struct ref_store *refs,\n+\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err);\n+\n typedef int ref_transaction_prepare_fn(struct ref_store *refs,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n@@ -670,6 +674,7 @@ struct ref_storage_be {\n \tref_store_init_fn *init;\n \tref_init_db_fn *init_db;\n \n+\tref_transaction_begin_fn *transaction_begin;\n \tref_transaction_prepare_fn *transaction_prepare;\n \tref_transaction_finish_fn *transaction_finish;\n \tref_transaction_abort_fn *transaction_abort;\n-- \ngitgitgadget\n\n"},{"id":"481969","messageId":"a5cb3adc6627622aa8a9eb0605da36429efa5466.1695059978.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","subject":"[PATCH 2/3] refs: move is_packed_transaction_needed out of packed-backend.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-18T17:59:37Z","receivedAt":"2023-09-18T17:59:53Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nIt is no longer specific to the packed backend.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/files-backend.c | 96 +++++++++++++++++++++++++++++++++++++++++++\n refs/packed-backend.c | 95 ------------------------------------------\n refs/packed-backend.h | 9 ----\n 3 files changed, 96 insertions(+), 104 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4a6781af783..c0a7e3d375b 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2630,6 +2630,101 @@ out:\n \treturn ret;\n }\n \n+\n+/*\n+ * Return true if `transaction` really needs to be carried out against\n+ * the specified packed_ref_store, or false if it can be skipped\n+ * (i.e., because it is an obvious NOOP). `ref_store` must be locked\n+ * before calling this function.\n+ */\n+static int is_packed_transaction_needed(struct ref_store *ref_store,\n+\t\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * We're only going to bother returning false for the common,\n+\t * trivial case that references are only being deleted, their\n+\t * old values are not being checked, and the old `packed-refs`\n+\t * file doesn't contain any of those reference(s). This gives\n+\t * false positives for some other cases that could\n+\t * theoretically be optimized away:\n+\t *\n+\t * 1. It could be that the old value is being verified without\n+\t * setting a new value. In this case, we could verify the\n+\t * old value here and skip the update if it agrees. If it\n+\t * disagrees, we could either let the update go through\n+\t * (the actual commit would re-detect and report the\n+\t * problem), or come up with a way of reporting such an\n+\t * error to *our* caller.\n+\t *\n+\t * 2. It could be that a new value is being set, but that it\n+\t * is identical to the current packed value of the\n+\t * reference.\n+\t *\n+\t * Neither of these cases will come up in the current code,\n+\t * because the only caller of this function passes to it a\n+\t * transaction that only includes `delete` updates with no\n+\t * `old_id`. Even if that ever changes, false positives only\n+\t * cause an optimization to be missed; they do not affect\n+\t * correctness.\n+\t */\n+\n+\t/*\n+\t * Start with the cheap checks that don't require old\n+\t * reference values to be read:\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\n+\t\tif (update->flags & REF_HAVE_OLD)\n+\t\t\t/* Have to check the old value -> needed. */\n+\t\t\treturn 1;\n+\n+\t\tif ((update->flags & REF_HAVE_NEW) && !is_null_oid(&update->new_oid))\n+\t\t\t/* Have to set a new value -> needed. */\n+\t\t\treturn 1;\n+\t}\n+\n+\t/*\n+\t * The transaction isn't checking any old values nor is it\n+\t * setting any nonzero new values, so it still might be able\n+\t * to be skipped. Now do the more expensive check: the update\n+\t * is needed if any of the updates is a delete, and the old\n+\t * `packed-refs` file contains a value for that reference.\n+\t */\n+\tret = 0;\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tint failure_errno;\n+\t\tunsigned int type;\n+\t\tstruct object_id oid;\n+\n+\t\tif (!(update->flags & REF_HAVE_NEW))\n+\t\t\t/*\n+\t\t\t * This reference isn't being deleted -> not\n+\t\t\t * needed.\n+\t\t\t */\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_read_raw_ref(ref_store, update->refname, &oid,\n+\t\t\t\t &referent, &type, &failure_errno) ||\n+\t\t failure_errno != ENOENT) {\n+\t\t\t/*\n+\t\t\t * We have to actually delete that reference\n+\t\t\t * -> this transaction is needed.\n+\t\t\t */\n+\t\t\tret = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n struct files_transaction_backend_data {\n \tstruct ref_transaction *packed_transaction;\n \tint packed_transaction_needed;\n@@ -3294,3 +3389,4 @@ struct ref_storage_be refs_be_files = {\n \t.delete_reflog = files_delete_reflog,\n \t.reflog_expire = files_reflog_expire\n };\n+\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 5df7fa8004f..ba895c845c0 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1455,101 +1455,6 @@ error:\n \treturn -1;\n }\n \n-int is_packed_transaction_needed(struct ref_store *ref_store,\n-\t\t\t\t struct ref_transaction *transaction)\n-{\n-\tstruct packed_ref_store *refs = packed_downcast(\n-\t\t\tref_store,\n-\t\t\tREF_STORE_READ,\n-\t\t\t\"is_packed_transaction_needed\");\n-\tstruct strbuf referent = STRBUF_INIT;\n-\tsize_t i;\n-\tint ret;\n-\n-\tif (!is_lock_file_locked(&refs->lock))\n-\t\tBUG(\"is_packed_transaction_needed() called while unlocked\");\n-\n-\t/*\n-\t * We're only going to bother returning false for the common,\n-\t * trivial case that references are only being deleted, their\n-\t * old values are not being checked, and the old `packed-refs`\n-\t * file doesn't contain any of those reference(s). This gives\n-\t * false positives for some other cases that could\n-\t * theoretically be optimized away:\n-\t *\n-\t * 1. It could be that the old value is being verified without\n-\t * setting a new value. In this case, we could verify the\n-\t * old value here and skip the update if it agrees. If it\n-\t * disagrees, we could either let the update go through\n-\t * (the actual commit would re-detect and report the\n-\t * problem), or come up with a way of reporting such an\n-\t * error to *our* caller.\n-\t *\n-\t * 2. It could be that a new value is being set, but that it\n-\t * is identical to the current packed value of the\n-\t * reference.\n-\t *\n-\t * Neither of these cases will come up in the current code,\n-\t * because the only caller of this function passes to it a\n-\t * transaction that only includes `delete` updates with no\n-\t * `old_id`. Even if that ever changes, false positives only\n-\t * cause an optimization to be missed; they do not affect\n-\t * correctness.\n-\t */\n-\n-\t/*\n-\t * Start with the cheap checks that don't require old\n-\t * reference values to be read:\n-\t */\n-\tfor (i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n-\n-\t\tif (update->flags & REF_HAVE_OLD)\n-\t\t\t/* Have to check the old value -> needed. */\n-\t\t\treturn 1;\n-\n-\t\tif ((update->flags & REF_HAVE_NEW) && !is_null_oid(&update->new_oid))\n-\t\t\t/* Have to set a new value -> needed. */\n-\t\t\treturn 1;\n-\t}\n-\n-\t/*\n-\t * The transaction isn't checking any old values nor is it\n-\t * setting any nonzero new values, so it still might be able\n-\t * to be skipped. Now do the more expensive check: the update\n-\t * is needed if any of the updates is a delete, and the old\n-\t * `packed-refs` file contains a value for that reference.\n-\t */\n-\tret = 0;\n-\tfor (i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n-\t\tint failure_errno;\n-\t\tunsigned int type;\n-\t\tstruct object_id oid;\n-\n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\t/*\n-\t\t\t * This reference isn't being deleted -> not\n-\t\t\t * needed.\n-\t\t\t */\n-\t\t\tcontinue;\n-\n-\t\tif (!refs_read_raw_ref(ref_store, update->refname, &oid,\n-\t\t\t\t &referent, &type, &failure_errno) ||\n-\t\t failure_errno != ENOENT) {\n-\t\t\t/*\n-\t\t\t * We have to actually delete that reference\n-\t\t\t * -> this transaction is needed.\n-\t\t\t */\n-\t\t\tret = 1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\tstrbuf_release(&referent);\n-\treturn ret;\n-}\n-\n struct packed_transaction_backend_data {\n \t/* True iff the transaction owns the packed-refs lock. */\n \tstruct string_list updates;\ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex ade3c8a5ac4..51a3b6a332a 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -17,13 +17,4 @@ struct ref_store *packed_ref_store_create(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags);\n \n-/*\n- * Return true if `transaction` really needs to be carried out against\n- * the specified packed_ref_store, or false if it can be skipped\n- * (i.e., because it is an obvious NOOP). `ref_store` must be locked\n- * before calling this function.\n- */\n-int is_packed_transaction_needed(struct ref_store *ref_store,\n-\t\t\t\t struct ref_transaction *transaction);\n-\n #endif /* REFS_PACKED_BACKEND_H */\n-- \ngitgitgadget\n\n"},{"id":"481970","messageId":"559c04cee998ed05c7b49bcac092df8a05cd709c.1695059978.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","subject":"[PATCH 3/3] refs: alternate reftable ref backend implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-18T17:59:38Z","receivedAt":"2023-09-18T17:59:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis introduces the reftable backend as an alternative to the packed\nbackend. This simplifies the code, because we now longer have to worry\nabout:\n\n- pseudorefs and the HEAD ref\n- worktrees\n- commands that blur together files and references (cherry-pick, rebase)\n\nThis deviates from the spec that in\nDocumentation/technical/reftable.txt. It might be possible to update\nthe code such that all writes by default go to reftable directly. Then\nthe result would be compatible with an implementation that writes only\nreftable (the reftable lock would still prevent races) relative to an\nimplementation that disregards loose refs. Or, JGit could be adapted\nto follow this implementation.\n\nFor this incremental path, the reftable format is arguably more\ncomplex than necessary, as\n\n- packed-refs doesn't support symrefs\n- reflogs aren't moved into reftable/\n\non the other hand, the code is already there, and it's well-structured\nand well-tested.\n\nThis implementation is a prototype. To test, you need to do `export\nGIT_TEST_REFTABLE=1`. Doing so creates a handful of errors in the\ntest-suite, most seemingly related to the new behavior of pack-refs\n(which creates a reftable/ dir and not a packed-refs file.), but it\nseems overseeable.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n config.mak.uname | 2 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs/files-backend.c | 8 +-\n refs/packed-backend.c | 14 +-\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1745 +++++++++++++++++++++++++++++++\n refs/reftable-backend.h | 8 +\n 8 files changed, 1766 insertions(+), 15 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 refs/reftable-backend.h\n\ndiff --git a/Makefile b/Makefile\nindex 57763093653..272d3f7f1e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1118,6 +1118,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 3bb03f423a0..843829c02fd 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -743,7 +743,7 @@ vcxproj:\n \n \t# Make .vcxproj files and add them\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a5215..989197aace0 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex c0a7e3d375b..725321a229e 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -9,6 +9,7 @@\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n #include \"packed-backend.h\"\n+#include \"reftable-backend.h\"\n #include \"../ident.h\"\n #include \"../iterator.h\"\n #include \"../dir-iterator.h\"\n@@ -103,8 +104,13 @@ static struct ref_store *files_ref_store_create(struct repository *repo,\n \trefs->store_flags = flags;\n \tget_common_dir_noenv(&sb, gitdir);\n \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n+\n+\t/* TODO: should look at the repo to decide whether to use packed-refs or\n+\t * reftable. */\n \trefs->packed_ref_store =\n-\t\tpacked_ref_store_create(repo, refs->gitcommondir, flags);\n+\t\tgit_env_bool(\"GIT_TEST_REFTABLE\", 0) \n+\t\t? git_reftable_ref_store_create(repo, refs->gitcommondir, flags)\n+\t\t: packed_ref_store_create(repo, refs->gitcommondir, flags);\n \n \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\n \tchdir_notify_reparent(\"files-backend $GIT_COMMONDIR\",\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex ba895c845c0..ad72cad8024 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1153,7 +1153,7 @@ static int write_packed_entry(FILE *fh, const char *refname,\n \treturn 0;\n }\n \n-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n+static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n {\n \tstruct packed_ref_store *refs =\n \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n@@ -1212,7 +1212,7 @@ int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n \treturn 0;\n }\n \n-void packed_refs_unlock(struct ref_store *ref_store)\n+static void packed_refs_unlock(struct ref_store *ref_store)\n {\n \tstruct packed_ref_store *refs = packed_downcast(\n \t\t\tref_store,\n@@ -1224,16 +1224,6 @@ void packed_refs_unlock(struct ref_store *ref_store)\n \trollback_lock_file(&refs->lock);\n }\n \n-int packed_refs_is_locked(struct ref_store *ref_store)\n-{\n-\tstruct packed_ref_store *refs = packed_downcast(\n-\t\t\tref_store,\n-\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n-\t\t\t\"packed_refs_is_locked\");\n-\n-\treturn is_lock_file_locked(&refs->lock);\n-}\n-\n /*\n * The packed-refs header line that we write out. Perhaps other traits\n * will be added later.\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 87f8e8b51d6..172ae23534b 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -700,6 +700,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..367bcae2857\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1745 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../environment.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+#include \"reftable-backend.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\n+\tstruct reftable_write_options write_options;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\treturn store->main_stack;\n+}\n+\n+static int should_log(const char *refname)\n+{\n+\treturn log_all_ref_updates != LOG_REFS_NONE &&\n+\t (log_all_ref_updates == LOG_REFS_ALWAYS ||\n+\t\tlog_all_ref_updates == LOG_REFS_UNSET ||\n+\t\tshould_autocreate_reflog(refname));\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *stripped;\n+\tparse_worktree_ref(ref, NULL, NULL, &stripped);\n+\treturn stripped;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type, int *failure_errno);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/* when we write log records, the hashes are owned by a struct\n+\t\t * oid */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+struct ref_store *git_reftable_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n+\t\t\t\t\t\tunsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint shared = get_shared_repository();\n+\tif (shared < 0)\n+\t\tshared = -shared;\n+\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = the_hash_algo->format_id;\n+\tif (shared && (shared & 0600))\n+\t\trefs->write_options.default_permissions = shared;\n+\n+\t/* XXX should this use `path` or `gitdir.buf` ? */\n+\tbase_ref_store_init(ref_store, repo, gitdir, &refs_be_reftable);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\trefs->base.repo = repo;\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err = reftable_new_stack(&refs->main_stack, refs->reftable_dir,\n+\t\t\t\t refs->write_options);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&sb);\n+\n+\t/* TODO something with chdir_notify_reparent() ? */\n+\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ri->ref_store;\n+\n+\twhile (ri->err == 0) {\n+\t\tint signed_flags = 0;\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tri->base.flags = 0;\n+\n+\t\tif (!strcmp(ri->ref.refname, \"HEAD\")) {\n+\t\t\t/*\n+\t\t\t HEAD should not be produced by default.Other\n+\t\t\t pseudorefs (FETCH_HEAD etc.) shouldn't be\n+\t\t\t stored in reftables at all.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(ri->base.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tif (ri->flags & DO_FOR_EACH_INCLUDE_BROKEN &&\n+\t\t check_refname_format(ri->base.refname,\n+\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\t/* This is odd, as REF_BAD_NAME and REF_ISBROKEN are\n+\t\t\t orthogonal, but it's what the spec says and the\n+\t\t\t files-backend does. */\n+\t\t\tri->base.flags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tri->base.flags = REF_ISSYMREF;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, refs->base.repo,\n+\t\t\t\t\t ri->base.oid, ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Arguably, resolving recursively following symlinks should be\n+\t\t * lifted to refs.c because it is shared between reftable and\n+\t\t * the files backend, but it's here now.\n+\t\t */\n+\t\tif (!refs_resolve_ref_unsafe(ri->ref_store, ri->ref.refname,\n+\t\t\t\t\t RESOLVE_REF_READING, &ri->oid,\n+\t\t\t\t\t &signed_flags)) {\n+\t\t\tri->base.flags = signed_flags;\n+\t\t\tif (ri->ref.value_type == REFTABLE_REF_SYMREF &&\n+\t\t\t ri->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (ri->ref.value_type == REFTABLE_REF_SYMREF &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (is_null_oid(&ri->oid)) {\n+\t\t\t\toidclr(&ri->oid);\n+\t\t\t\tri->base.flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t}\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tconst char **exclude_patterns_TODO,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\t\tint failure_errno;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type, &failure_errno);\n+\t\tif (err < 0 && failure_errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs,\n+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n+\tint i;\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n+\t\t !(u->flags & REF_LOG_ONLY)) {\n+\t\t\tstruct object *o =\n+\t\t\t\tparse_object(refs->base.repo, &u->new_oid);\n+\t\t\tif (!o) {\n+\t\t\t\tstrbuf_addf(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t\"trying to write ref '%s' with nonexistent object %s\",\n+\t\t\t\t\tu->refname, oid_to_hex(&u->new_oid));\n+\t\t\t\terr = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\tif (add) {\n+\t\t\treftable_addition_destroy(add);\n+\t\t\tadd = NULL;\n+\t\t}\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tif (!errbuf->len)\n+\t\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\n+\t/* XXX. Shouldn't this be handled generically in refs.c? */\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != !resolved) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tint log_count = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[log_count];\n+\t\tstruct object_id old_id = *null_oid();\n+\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tif ((u->flags & REF_FORCE_CREATE_REFLOG) ||\n+\t\t should_log(u->refname))\n+\t\t\tlog_count++;\n+\t\tfill_reftable_log_record(log);\n+\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = xstrdup(u->refname);\n+\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\tlog->value.update.message =\n+\t\t\txstrndup(u->msg, refs->write_options.block_size / 2);\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\t\tlog->value.update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < log_count; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = xstrdup(arg->refnames->items[i].string);\n+\t\tif (!should_log(log.refname)) {\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlog.value.update.message = xstrndup(\n+\t\t\targ->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.value.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.value.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = xstrdup(create->refname);\n+\t\tif (!should_log(log.refname)) {\n+\t\t\treturn err;\n+\t\t}\n+\t\tlog.update_index = ts;\n+\t\tlog.value.update.message =\n+\t\t\txstrndup(create->logmsg,\n+\t\t\t\t create->refs->write_options.block_size / 2);\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL)) {\n+\t\t\tlog.value.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL)) {\n+\t\t\tlog.value.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.value.update.old_hash ||\n+\t\t log.value.update.new_hash) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.value.update.message = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tstruct reftable_ref_record todo[2] = {\n+\t\t{\n+\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t.update_index = ts,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t},\n+\t\told_ref,\n+\t};\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (!reftable_stack_read_ref(arg->stack, arg->newname, &new_ref)) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\ttodo[1].update_index = ts;\n+\ttodo[1].refname = (char *)arg->newname;\n+\n+\terr = reftable_writer_add_refs(writer, todo, 2);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tint firstlog = 0;\n+\t\tint lastlog = 2;\n+\t\tchar *msg = xstrndup(arg->logmsg,\n+\t\t\t\t arg->refs->write_options.block_size / 2);\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = xstrdup(arg->oldname);\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].value.update.message = msg;\n+\t\ttodo[0].value.update.old_hash = val1;\n+\t\ttodo[0].value.update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = xstrdup(arg->newname);\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].value.update.old_hash = NULL;\n+\t\ttodo[1].value.update.new_hash = val1;\n+\t\ttodo[1].value.update.message = xstrdup(msg);\n+\n+\t\tif (!should_log(todo[1].refname)) {\n+\t\t\tlastlog--;\n+\t\t}\n+\t\tif (!should_log(todo[0].refname)) {\n+\t\t\tfirstlog++;\n+\t\t}\n+\t\terr = reftable_writer_add_logs(writer, &todo[firstlog],\n+\t\t\t\t\t lastlog - firstlog);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX what should we write into the reflog if we rename a\n+\t\t * symref? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX this copies the entire reflog history. Is this the right\n+\t * semantics? should clear out existing reflog entries for oldname? */\n+\tif (!should_log(arg->newname))\n+\t\tgoto done;\n+\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tstruct git_reftable_ref_store *refs;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint flags = 0;\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!refs_resolve_ref_unsafe(&ri->refs->base, ri->log.refname,\n+\t\t\t\t\t 0, &ri->oid, &flags)) {\n+\t\t\terror(\"bad ref for %s\", ri->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\tri->base.oid = &ri->oid;\n+\t\tri->base.flags = flags;\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct reftable_stack *stack = refs->main_stack;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(stack);\n+\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\n+\tri->refs = refs;\n+\tif (err < 0) {\n+\t\tfree(ri);\n+\t\t/* XXX how to handle errors in iterator_begin()? */\n+\t\treturn NULL;\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\t/* placeholder for existence. */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfull_committer = fmt_ident(log.value.update.name,\n+\t\t\t\t\t log.value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log.value.update.time, log.value.update.tz_offset,\n+\t\t\t log.value.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->value.update.old_hash);\n+\t\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\t/* placeholder for existence. */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfull_committer = fmt_ident(log->value.update.name,\n+\t\t\t\t\t log->value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t\t log->value.update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *argv)\n+{\n+\tstruct write_reflog_existence_arg *arg =\n+\t\t(struct write_reflog_existence_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tint err = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (err <= 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tlog.refname = (char *)arg->refname;\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\terr = reftable_writer_add_log(writer, &log);\n+\n+\t/* field is not malloced */\n+\tlog.refname = NULL;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn err;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (log.value_type == REFTABLE_LOG_DELETION)\n+\t\t\tcontinue;\n+\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\tint live_records = 0;\n+\tuint64_t max_ts = 0;\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\t\tif (max_ts < arg->records[i].update_index)\n+\t\t\tmax_ts = arg->records[i].update_index;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tif (live_records == 0) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = max_ts + 1,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t/* existence dummy has null new/old oid */\n+\t\t};\n+\t\tint err;\n+\t\tif (log.update_index < ts)\n+\t\t\tlog.update_index = ts;\n+\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_reflog_expire(\n+\tstruct ref_store *ref_store, const char *refname, unsigned int flags,\n+\treflog_expiry_prepare_fn prepare_fn,\n+\treflog_expiry_should_prune_fn should_prune_fn,\n+\treflog_expiry_cleanup_fn cleanup_fn, void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\n+\t On the plus side, because we do the expiration per ref, we can easily\n+\t insert the reflog existence dummies.\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = { NULL };\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tstruct object_id oid = { 0 };\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tif (!reftable_stack_read_ref(stack, refname, &ref_record)) {\n+\t\tuint8_t *hash = reftable_ref_record_val1(&ref_record);\n+\t\tif (hash)\n+\t\t\toidread(&oid, hash);\n+\t}\n+\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].value.update.old_hash);\n+\t\toidread(&noid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash) {\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* future improvement: we could skip writing records that were\n+\t\t * not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type, int *failure_errno)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err == 0 && ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t} else {\n+\t\terr = -1;\n+\t}\n+\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.next = &refs_be_files,\n+\t.name = \"reftable\",\n+\t.init = git_reftable_ref_store_create,\n+\t.init_db = git_reftable_init_db,\n+\t.transaction_prepare = git_reftable_transaction_prepare,\n+\t.transaction_finish = git_reftable_transaction_finish,\n+\t.transaction_abort = git_reftable_transaction_abort,\n+\t.initial_transaction_commit = git_reftable_transaction_initial_commit,\n+\n+\t.pack_refs = git_reftable_pack_refs,\n+\t.create_symref = git_reftable_create_symref,\n+\t.delete_refs = git_reftable_delete_refs,\n+\t.rename_ref = git_reftable_rename_ref,\n+\t.copy_ref = git_reftable_copy_ref,\n+\n+\t.iterator_begin = git_reftable_ref_iterator_begin,\n+\t.read_raw_ref = git_reftable_read_raw_ref,\n+\t.read_symbolic_ref = git_reftable_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = git_reftable_reflog_iterator_begin,\n+\t.for_each_reflog_ent = git_reftable_for_each_reflog_ent_oldest_first,\n+\t.for_each_reflog_ent_reverse =\n+\t\tgit_reftable_for_each_reflog_ent_newest_first,\n+\t.reflog_exists = git_reftable_reflog_exists,\n+\t.create_reflog = git_reftable_create_reflog,\n+\t.delete_reflog = git_reftable_delete_reflog,\n+\t.reflog_expire = git_reftable_reflog_expire,\n+};\ndiff --git a/refs/reftable-backend.h b/refs/reftable-backend.h\nnew file mode 100644\nindex 00000000000..bf5fd42229a\n--- /dev/null\n+++ b/refs/reftable-backend.h\n@@ -0,0 +1,8 @@\n+#ifndef REFS_REFTABLE_BACKEND_H\n+#define REFS_REFTABLE_BACKEND_H\n+\n+struct ref_store *git_reftable_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n+\t\t\t\t\t\tunsigned int store_flags);\n+\n+#endif /* REFS_REFTABLE_BACKEND_H */\n-- \ngitgitgadget\n"},{"id":"481995","messageId":"xmqqfs3bje6f.fsf@gitster.g","threadId":"60240","inReplyTo":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] Simple reftable backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-18T22:43:04Z","receivedAt":"2023-09-18T22:43:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> I think the first part (\"refs: push lock management into packed backend\") is\n> a good improvement overall, and could be landed separately without much\n> tweaking.\n\nIt looked quite nice to remove more knowledge specific to the\n.git/packed-refs backend from the files backend, and instead have\nthe files backend rely more on the backend-provided methods to make\nit work together with whatever backend happens to be in use. There\nwere a few minor things I found puzzling, but other than that, the\ntwo patches looked mostly sensible.\n"},{"id":"481996","messageId":"xmqqa5tjje0y.fsf@gitster.g","threadId":"60240","inReplyTo":"dea0fbb139a82fe449a7edab6a8f445ce763d0c0.1695059978.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/3] refs: push lock management into packed backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-18T22:46:21Z","receivedAt":"2023-09-18T22:46:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> \tstruct ref_transaction *tr;\n> +\tint ret = 0;\n> \tassert(err);\n> \n> \tCALLOC_ARRAY(tr, 1);\n> \ttr->ref_store = refs;\n> +\n> +\tif (refs->be->transaction_begin)\n> +\t\tret = refs->be->transaction_begin(refs, tr, err);\n> +\tif (ret) {\n> +\t\tfree(tr);\n> +\t\treturn NULL;\n> +\t}\n> \treturn tr;\n> }\n\nThis looks a bit more convoluted than necessary. Is it the same as\n\n\tif (refs->be->transaction_begin &&\n\t refs->be->transaction_begin(refs, tr, err))\n\t\tFREE_AND_NULL(tr);\n\n\n> +\tif (backend_data->packed_transaction) { \n> +\t\tif (backend_data->packed_transaction_needed) {\n> +\t\t\tret = ref_transaction_commit(packed_transaction, err);\n> +\t\t\tif (ret)\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t/* TODO: leaks on error path. */\n> +\t\t\tref_transaction_free(packed_transaction);\n> +\t\t\tpacked_transaction = NULL;\n> +\t\t\tbackend_data->packed_transaction = NULL;\n> +\t\t} else {\n\nIf it were just a matter of flipping the early return and freeing of\nthe transaction before going to clean-up, then that would have been\nless effort than leaving the TODO: comment. What other things are\nneeded to plug this leak?\n"},{"id":"482016","messageId":"CAFQ2z_O-s-NT+DfxTSSuTq1Z6J0LZz8X8cFTPoOohfqX2rM-yw@mail.gmail.com","threadId":"60240","inReplyTo":"xmqqa5tjje0y.fsf@gitster.g","subject":"Re: [PATCH 1/3] refs: push lock management into packed backend","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2023-09-19T15:52:00Z","receivedAt":"2023-09-19T15:52:17Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Sep 19, 2023 at 12:46 AM Junio C Hamano wrote:\n> This looks a bit more convoluted than necessary. Is it the same as\n>\n> if (refs->be->transaction_begin &&\n> refs->be->transaction_begin(refs, tr, err))\n> FREE_AND_NULL(tr);\n\nChanged this in the next version.\n\n> > + /* TODO: leaks on error path. */\n> > + ref_transaction_free(packed_transaction);\n> > + packed_transaction = NULL;\n> > + backend_data->packed_transaction = NULL;\n> > + } else {\n>\n> If it were just a matter of flipping the early return and freeing of\n> the transaction before going to clean-up, then that would have been\n> less effort than leaving the TODO: comment. What other things are\n> needed to plug this leak?\n\nyou're not missing something. I didn't have enough time yesterday to\nlook into all the details.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Liana Sebastian\n"},{"id":"482055","messageId":"e99f3d2005633921299d4ee47f8c05b6f7e9391b.1695214968.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] refs: construct transaction using a _begin callback","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:43Z","receivedAt":"2023-09-20T13:02:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis prepares for taking locks on constructing the transaction.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs.c | 8 +++-----\n refs/debug.c | 8 ++++++++\n refs/files-backend.c | 8 ++++++++\n refs/packed-backend.c | 9 +++++++++\n refs/refs-internal.h | 4 ++++\n 5 files changed, 32 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex fcae5dddc60..f7ba3e56284 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1129,11 +1129,9 @@ int read_ref_at(struct ref_store *refs, const char *refname,\n struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,\n \t\t\t\t\t\t struct strbuf *err)\n {\n-\tstruct ref_transaction *tr;\n-\tassert(err);\n-\n-\tCALLOC_ARRAY(tr, 1);\n-\ttr->ref_store = refs;\n+\tstruct ref_transaction *tr = refs->be->transaction_begin(refs, err);\n+\tif (tr)\n+\t\ttr->ref_store = refs;\n \treturn tr;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex b7ffc4ce67e..95fb3eb4430 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -41,6 +41,13 @@ static int debug_init_db(struct ref_store *refs, struct strbuf *err)\n \treturn res;\n }\n \n+static struct ref_transaction *debug_transaction_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t struct strbuf *err) {\n+\tstruct ref_transaction *tr;\n+\tCALLOC_ARRAY(tr, 1);\n+\treturn tr;\n+}\n+\n static int debug_transaction_prepare(struct ref_store *refs,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n@@ -451,6 +458,7 @@ struct ref_storage_be refs_be_debug = {\n \t * has a function we should also have a wrapper for it here.\n \t * Test the output with \"GIT_TRACE_REFS=1\".\n \t */\n+\t.transaction_begin = debug_transaction_begin,\n \t.transaction_prepare = debug_transaction_prepare,\n \t.transaction_finish = debug_transaction_finish,\n \t.transaction_abort = debug_transaction_abort,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 341354182bb..d83eea8b408 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2681,6 +2681,13 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n }\n \n+static struct ref_transaction *files_transaction_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t struct strbuf *err) {\n+\tstruct ref_transaction *tr;\n+\tCALLOC_ARRAY(tr, 1);\n+\treturn tr;\n+}\n+\n static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n@@ -3297,6 +3304,7 @@ struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_create,\n \t.init_db = files_init_db,\n+\t.transaction_begin = files_transaction_begin,\n \t.transaction_prepare = files_transaction_prepare,\n \t.transaction_finish = files_transaction_finish,\n \t.transaction_abort = files_transaction_abort,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 59c78d7941f..4f5b1025605 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1580,6 +1580,14 @@ static void packed_transaction_cleanup(struct packed_ref_store *refs,\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n }\n \n+\n+static struct ref_transaction *packed_transaction_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t\tstruct strbuf *err) {\n+\tstruct ref_transaction *tr;\n+\tCALLOC_ARRAY(tr, 1);\n+\treturn tr;\n+}\n+\n static int packed_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err)\n@@ -1758,6 +1766,7 @@ struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_create,\n \t.init_db = packed_init_db,\n+\t.transaction_begin = packed_transaction_begin,\n \t.transaction_prepare = packed_transaction_prepare,\n \t.transaction_finish = packed_transaction_finish,\n \t.transaction_abort = packed_transaction_abort,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 9db8aec4da8..0a15e8a2ac8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -531,6 +531,9 @@ typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n \n typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n \n+typedef struct ref_transaction *ref_transaction_begin_fn(struct ref_store *refs,\n+\t\t\t\t\t\t\t struct strbuf *err);\n+\n typedef int ref_transaction_prepare_fn(struct ref_store *refs,\n \t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t struct strbuf *err);\n@@ -670,6 +673,7 @@ struct ref_storage_be {\n \tref_store_init_fn *init;\n \tref_init_db_fn *init_db;\n \n+\tref_transaction_begin_fn *transaction_begin;\n \tref_transaction_prepare_fn *transaction_prepare;\n \tref_transaction_finish_fn *transaction_finish;\n \tref_transaction_abort_fn *transaction_abort;\n-- \ngitgitgadget\n\n"},{"id":"482056","messageId":"9a459259330515d5b127e141c007ebc01d124428.1695214968.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] refs: wrap transaction in a debug-specific transaction","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:44Z","receivedAt":"2023-09-20T13:03:02Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis ensures that all transaction methods are properly printed. This is\nneeded because the callbacks are called as\ntransaction->ref_store->backend->$CALLBACK.\n\nOriginally, there was a hacky workaround which overwrote\ntransaction->ref_store before passing it into the non-debug prepare()\ncallback. This would print something for the prepare function, but not\nfor abort/finish functions.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/debug.c | 83 +++++++++++++++++++++++++++++++++++++---------------\n 1 file changed, 59 insertions(+), 24 deletions(-)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 95fb3eb4430..30509da4b89 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -43,25 +43,18 @@ static int debug_init_db(struct ref_store *refs, struct strbuf *err)\n \n static struct ref_transaction *debug_transaction_begin(struct ref_store *ref_store,\n \t\t\t\t\t\t struct strbuf *err) {\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_transaction *tr;\n+\tstruct ref_transaction *sub_tr = ref_store_transaction_begin(drefs->refs, err);\n+\ttrace_printf_key(&trace_refs, \"transaction_begin: %s\\n\", err->buf);\n+\tif (!sub_tr) {\n+\t\treturn NULL;\n+\t}\n \tCALLOC_ARRAY(tr, 1);\n+\ttr->backend_data = sub_tr;\n \treturn tr;\n }\n \n-static int debug_transaction_prepare(struct ref_store *refs,\n-\t\t\t\t struct ref_transaction *transaction,\n-\t\t\t\t struct strbuf *err)\n-{\n-\tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n-\tint res;\n-\ttransaction->ref_store = drefs->refs;\n-\tres = drefs->refs->be->transaction_prepare(drefs->refs, transaction,\n-\t\t\t\t\t\t err);\n-\ttrace_printf_key(&trace_refs, \"transaction_prepare: %d \\\"%s\\\"\\n\", res,\n-\t\t\t err->buf);\n-\treturn res;\n-}\n-\n static void print_update(int i, const char *refname,\n \t\t\t const struct object_id *old_oid,\n \t\t\t const struct object_id *new_oid, unsigned int flags,\n@@ -77,20 +70,59 @@ static void print_update(int i, const char *refname,\n \ttype &= 0xf; /* see refs.h REF_* */\n \tflags &= REF_HAVE_NEW | REF_HAVE_OLD | REF_NO_DEREF |\n \t\tREF_FORCE_CREATE_REFLOG;\n-\ttrace_printf_key(&trace_refs, \"%d: %s %s -> %s (F=0x%x, T=0x%x) \\\"%s\\\"\\n\", i, refname,\n+\ttrace_printf_key(&trace_refs, \" %d: '%s' %s -> %s (F=0x%x, T=0x%x) \\\"%s\\\"\\n\", i, refname,\n \t\to, n, flags, type, msg);\n }\n \n-static void print_transaction(struct ref_transaction *transaction)\n+static void print_transaction(struct ref_transaction *transaction,\n+\t\t\t const char *action,\n+\t\t\t int res, struct strbuf *err)\n {\n \tint i;\n-\ttrace_printf_key(&trace_refs, \"transaction {\\n\");\n+\ttrace_printf_key(&trace_refs, \"transaction %s {\\n\", action);\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *u = transaction->updates[i];\n \t\tprint_update(i, u->refname, &u->old_oid, &u->new_oid, u->flags,\n \t\t\t u->type, u->msg);\n \t}\n-\ttrace_printf_key(&trace_refs, \"}\\n\");\n+\ttrace_printf_key(&trace_refs, \"}: %d '%s'\\n\", res, err->buf);\n+}\n+\n+static void copy_update(struct ref_update *dst,\n+\t\t\tstruct ref_update *src) {\n+\t/* dst->refname is const char; can't assign structs */\n+\tdst->new_oid = src->new_oid;\n+\tdst->old_oid = src->old_oid;\n+\tdst->flags = src->flags;\n+\tdst->backend_data = NULL;\n+\tdst->type = src->type;\n+\tdst->msg = xstrdup(src->msg);\n+\tdst->parent_update = NULL;\n+\t/* refname is done as part of FLEX_ALLOC_STR. */\n+}\n+\n+static int debug_transaction_prepare(struct ref_store *refs,\n+\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n+\tstruct ref_transaction *sub_transaction = transaction->backend_data;\n+\tint res;\n+\tint i;\n+\n+\tALLOC_GROW(sub_transaction->updates, transaction->nr, sub_transaction->alloc);\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *up = transaction->updates[i];\n+\t\tstruct ref_update *sub_up;\n+\t\tFLEX_ALLOC_STR(sub_up, refname, up->refname);\n+\t\tcopy_update(sub_up, up);\n+\t\tsub_transaction->updates[i] = sub_up;\n+\t}\n+\tsub_transaction->nr = transaction->nr;\n+\tres = drefs->refs->be->transaction_prepare(drefs->refs, sub_transaction,\n+\t\t\t\t\t\t err);\n+\tprint_transaction(sub_transaction, \"prepare\", res, err);\n+\treturn res;\n }\n \n static int debug_transaction_finish(struct ref_store *refs,\n@@ -98,12 +130,11 @@ static int debug_transaction_finish(struct ref_store *refs,\n \t\t\t\t struct strbuf *err)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n+\tstruct ref_transaction *sub_transaction = transaction->backend_data;\n \tint res;\n-\ttransaction->ref_store = drefs->refs;\n-\tres = drefs->refs->be->transaction_finish(drefs->refs, transaction,\n+\tres = drefs->refs->be->transaction_finish(drefs->refs, sub_transaction,\n \t\t\t\t\t\t err);\n-\tprint_transaction(transaction);\n-\ttrace_printf_key(&trace_refs, \"finish: %d\\n\", res);\n+\tprint_transaction(sub_transaction, \"finish\", res, err);\n \treturn res;\n }\n \n@@ -112,9 +143,11 @@ static int debug_transaction_abort(struct ref_store *refs,\n \t\t\t\t struct strbuf *err)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n+\tstruct ref_transaction *sub_transaction = transaction->backend_data;\n \tint res;\n \ttransaction->ref_store = drefs->refs;\n-\tres = drefs->refs->be->transaction_abort(drefs->refs, transaction, err);\n+\tres = drefs->refs->be->transaction_abort(drefs->refs, sub_transaction, err);\n+\tprint_transaction(sub_transaction, \"abort\", res, err);\n \treturn res;\n }\n \n@@ -123,10 +156,12 @@ static int debug_initial_transaction_commit(struct ref_store *refs,\n \t\t\t\t\t struct strbuf *err)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n+\tstruct ref_transaction *sub_transaction = transaction->backend_data;\n \tint res;\n \ttransaction->ref_store = drefs->refs;\n \tres = drefs->refs->be->initial_transaction_commit(drefs->refs,\n-\t\t\t\t\t\t\t transaction, err);\n+\t\t\t\t\t\t\t sub_transaction, err);\n+\tprint_transaction(sub_transaction, \"commit\", res, err);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"482057","messageId":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.git.git.1695059978.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] RFC: simple reftable backend","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:42Z","receivedAt":"2023-09-20T13:03:05Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This series comes from a conversation with Patrick Steinhardt at Gitlab, who\nhave an interest in a more scalable ref storage system.\n\nI unfortunately don't have business reasons to spend much time on this\nproject anymore, and the original, sweeping migration has too many open\nquestions.\n\nI thought of an alternate approach, and I wanted to show it as input to\ndiscussions at the contributor summit.\n\nI think the first part (everything before \"refs: alternate reftable ref\nbackend implementation\") is a good improvement overall, and could be landed\nseparately without much tweaking.\n\nThe second part (\"refs: alternate reftable ref backend implementation\" and\nfollow-on) is something open for discussion: the alternative \"packed\nstorage\" is based on reftable, but it could conceivably be a different type\nof database/file format too. I think it's attractive to use reftable here,\nbecause over time more data (symrefs, reflog) could be moved into reftable.\n\nI'm offering this series as a RFC. I hope that someone else (Patrick?) can\nlook into getting the bits and pieces of this merged.\n\nHan-Wen Nienhuys (6):\n refs: construct transaction using a _begin callback\n refs: wrap transaction in a debug-specific transaction\n refs: push lock management into packed backend\n refs: move is_packed_transaction_needed out of packed-backend.c\n refs: alternate reftable ref backend implementation\n refs: always try to do packed transactions for reftable\n\n Makefile | 1 +\n config.mak.uname | 2 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs.c | 13 +-\n refs/debug.c | 87 +-\n refs/files-backend.c | 216 ++--\n refs/packed-backend.c | 149 +--\n refs/packed-backend.h | 19 -\n refs/refs-internal.h | 5 +\n refs/reftable-backend.c | 1658 +++++++++++++++++++++++++++++++\n refs/reftable-backend.h | 8 +\n 11 files changed, 1929 insertions(+), 231 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 refs/reftable-backend.h\n\n\nbase-commit: bda494f4043963b9ec9a1ecd4b19b7d1cd9a0518\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1574%2Fhanwen%2Fsimple-reftable-backend-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1574/hanwen/simple-reftable-backend-v2\nPull-Request: https://github.com/git/git/pull/1574\n\nRange-diff vs v1:\n\n -: ----------- > 1: e99f3d20056 refs: construct transaction using a _begin callback\n -: ----------- > 2: 9a459259330 refs: wrap transaction in a debug-specific transaction\n 1: dea0fbb139a ! 3: 8dedb23eb69 refs: push lock management into packed backend\n @@ Metadata\n ## Commit message ##\n refs: push lock management into packed backend\n \n - Introduces a ref backend method transaction_begin, which for the\n - packed backend takes packed-refs.lock.\n + Take packed-refs.lock in the transaction_begin of the packed backend.\n \n This decouples the files backend from the packed backend, allowing the\n latter to be swapped out by another ref backend.\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## refs.c ##\n -@@ refs.c: struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,\n - \t\t\t\t\t\t struct strbuf *err)\n - {\n - \tstruct ref_transaction *tr;\n -+\tint ret = 0;\n - \tassert(err);\n - \n - \tCALLOC_ARRAY(tr, 1);\n - \ttr->ref_store = refs;\n -+\n -+\tif (refs->be->transaction_begin)\n -+\t\tret = refs->be->transaction_begin(refs, tr, err);\n -+\tif (ret) {\n -+\t\tfree(tr);\n -+\t\treturn NULL;\n -+\t}\n - \treturn tr;\n - }\n - \n -\n - ## refs/debug.c ##\n -@@ refs/debug.c: static int debug_init_db(struct ref_store *refs, struct strbuf *err)\n - \treturn res;\n - }\n - \n -+static int debug_transaction_begin(struct ref_store *refs,\n -+\t\t\t\t struct ref_transaction *transaction,\n -+\t\t\t\t struct strbuf *err)\n -+{\n -+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n -+\tint res;\n -+\ttransaction->ref_store = drefs->refs;\n -+\tres = drefs->refs->be->transaction_begin(drefs->refs, transaction,\n -+\t\t\t\t\t\t err);\n -+\ttrace_printf_key(&trace_refs, \"transaction_begin: %d \\\"%s\\\"\\n\", res,\n -+\t\t\t err->buf);\n -+\treturn res;\n -+}\n -+\n - static int debug_transaction_prepare(struct ref_store *refs,\n - \t\t\t\t struct ref_transaction *transaction,\n - \t\t\t\t struct strbuf *err)\n -@@ refs/debug.c: struct ref_storage_be refs_be_debug = {\n - \t * has a function we should also have a wrapper for it here.\n - \t * Test the output with \"GIT_TRACE_REFS=1\".\n - \t */\n -+\t.transaction_begin = debug_transaction_begin,\n - \t.transaction_prepare = debug_transaction_prepare,\n - \t.transaction_finish = debug_transaction_finish,\n - \t.transaction_abort = debug_transaction_abort,\n +@@ refs.c: int ref_transaction_abort(struct ref_transaction *transaction,\n + \tint ret = 0;\n + \n + \tswitch (transaction->state) {\n +-\tcase REF_TRANSACTION_OPEN:\n +-\t\t/* No need to abort explicitly. */\n +-\t\tbreak;\n + \tcase REF_TRANSACTION_PREPARED:\n ++\tcase REF_TRANSACTION_OPEN:\n ++\t\t/* an OPEN transactions may have a lock. */\n + \t\tret = refs->be->transaction_abort(refs, transaction, err);\n + \t\tbreak;\n + \tcase REF_TRANSACTION_CLOSED:\n \n ## refs/files-backend.c ##\n @@ refs/files-backend.c: static int files_pack_refs(struct ref_store *ref_store,\n @@ refs/files-backend.c: static int files_transaction_finish(struct ref_store *ref_\n -\t\tbackend_data->packed_transaction = NULL;\n -\t\tif (ret)\n -\t\t\tgoto cleanup;\n -+\tif (backend_data->packed_transaction) { \n ++\tif (backend_data->packed_transaction) {\n +\t\tif (backend_data->packed_transaction_needed) {\n +\t\t\tret = ref_transaction_commit(packed_transaction, err);\n +\t\t\tif (ret)\n +\t\t\t\tgoto cleanup;\n -+\t\t\t/* TODO: leaks on error path. */\n ++\n +\t\t\tref_transaction_free(packed_transaction);\n +\t\t\tpacked_transaction = NULL;\n +\t\t\tbackend_data->packed_transaction = NULL;\n @@ refs/files-backend.c: static int files_initial_transaction_commit(struct ref_sto\n \t\tref_transaction_free(packed_transaction);\n \n ## refs/packed-backend.c ##\n +@@ refs/packed-backend.c: static int write_packed_entry(FILE *fh, const char *refname,\n + \treturn 0;\n + }\n + \n +-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n ++static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n + {\n + \tstruct packed_ref_store *refs =\n + \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n +@@ refs/packed-backend.c: int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n + \treturn 0;\n + }\n + \n +-void packed_refs_unlock(struct ref_store *ref_store)\n ++static void packed_refs_unlock(struct ref_store *ref_store)\n + {\n + \tstruct packed_ref_store *refs = packed_downcast(\n + \t\t\tref_store,\n +@@ refs/packed-backend.c: void packed_refs_unlock(struct ref_store *ref_store)\n + \trollback_lock_file(&refs->lock);\n + }\n + \n +-int packed_refs_is_locked(struct ref_store *ref_store)\n +-{\n +-\tstruct packed_ref_store *refs = packed_downcast(\n +-\t\t\tref_store,\n +-\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n +-\t\t\t\"packed_refs_is_locked\");\n +-\n +-\treturn is_lock_file_locked(&refs->lock);\n +-}\n +-\n + /*\n + * The packed-refs header line that we write out. Perhaps other traits\n + * will be added later.\n @@ refs/packed-backend.c: int is_packed_transaction_needed(struct ref_store *ref_store,\n \n struct packed_transaction_backend_data {\n @@ refs/packed-backend.c: static void packed_transaction_cleanup(struct packed_ref_\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n }\n \n -+static int packed_transaction_begin(struct ref_store *ref_store,\n -+\t\t\t\t struct ref_transaction *transaction,\n +-\n + static struct ref_transaction *packed_transaction_begin(struct ref_store *ref_store,\n +-\t\t\t\t\t\t\tstruct strbuf *err) {\n +-\tstruct ref_transaction *tr;\n +-\tCALLOC_ARRAY(tr, 1);\n +-\treturn tr;\n +\t\t\t\t struct strbuf *err)\n +{\n +\tstruct packed_ref_store *refs = packed_downcast(\n @@ refs/packed-backend.c: static void packed_transaction_cleanup(struct packed_ref_\n +\t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n +\t\t\t\"ref_transaction_begin\");\n +\tstruct packed_transaction_backend_data *data;\n -+\n -+\tCALLOC_ARRAY(data, 1);\n -+\tstring_list_init_nodup(&data->updates);\n ++\tstruct ref_transaction *transaction;\n +\n +\tif (!is_lock_file_locked(&refs->lock)) {\n +\t\tif (packed_refs_lock(ref_store, 0, err))\n -+\t\t\t/* todo: leaking data */\n -+\t\t\treturn -1;\n ++\t\t\treturn NULL;\n +\t}\n ++\tCALLOC_ARRAY(transaction, 1);\n ++\tCALLOC_ARRAY(data, 1);\n ++\tstring_list_init_nodup(&data->updates);\n +\ttransaction->backend_data = data;\n -+\treturn 0;\n -+}\n -+\n ++\treturn transaction;\n + }\n + \n static int packed_transaction_prepare(struct ref_store *ref_store,\n - \t\t\t\t struct ref_transaction *transaction,\n - \t\t\t\t struct strbuf *err)\n @@ refs/packed-backend.c: static int packed_transaction_prepare(struct ref_store *ref_store,\n \t\t\tref_store,\n \t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n @@ refs/packed-backend.c: static int packed_transaction_prepare(struct ref_store *r\n \tif (write_with_updates(refs, &data->updates, err))\n \t\tgoto failure;\n \n -@@ refs/packed-backend.c: struct ref_storage_be refs_be_packed = {\n - \t.name = \"packed\",\n - \t.init = packed_ref_store_create,\n - \t.init_db = packed_init_db,\n -+\t.transaction_begin = packed_transaction_begin,\n - \t.transaction_prepare = packed_transaction_prepare,\n - \t.transaction_finish = packed_transaction_finish,\n - \t.transaction_abort = packed_transaction_abort,\n \n ## refs/packed-backend.h ##\n @@ refs/packed-backend.h: struct ref_store *packed_ref_store_create(struct repository *repo,\n @@ refs/packed-backend.h: struct ref_store *packed_ref_store_create(struct reposito\n /*\n * Return true if `transaction` really needs to be carried out against\n * the specified packed_ref_store, or false if it can be skipped\n -\n - ## refs/refs-internal.h ##\n -@@ refs/refs-internal.h: typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n - \n - typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n - \n -+typedef int ref_transaction_begin_fn(struct ref_store *refs,\n -+\t\t\t\t struct ref_transaction *transaction,\n -+\t\t\t\t struct strbuf *err);\n -+\n - typedef int ref_transaction_prepare_fn(struct ref_store *refs,\n - \t\t\t\t struct ref_transaction *transaction,\n - \t\t\t\t struct strbuf *err);\n -@@ refs/refs-internal.h: struct ref_storage_be {\n - \tref_store_init_fn *init;\n - \tref_init_db_fn *init_db;\n - \n -+\tref_transaction_begin_fn *transaction_begin;\n - \tref_transaction_prepare_fn *transaction_prepare;\n - \tref_transaction_finish_fn *transaction_finish;\n - \tref_transaction_abort_fn *transaction_abort;\n 2: a5cb3adc662 ! 4: 0b8919b05c4 refs: move is_packed_transaction_needed out of packed-backend.c\n @@ refs/files-backend.c: out:\n \treturn ret;\n }\n \n -+\n +/*\n + * Return true if `transaction` really needs to be carried out against\n + * the specified packed_ref_store, or false if it can be skipped\n @@ refs/files-backend.c: out:\n struct files_transaction_backend_data {\n \tstruct ref_transaction *packed_transaction;\n \tint packed_transaction_needed;\n -@@ refs/files-backend.c: struct ref_storage_be refs_be_files = {\n - \t.delete_reflog = files_delete_reflog,\n - \t.reflog_expire = files_reflog_expire\n - };\n -+\n \n ## refs/packed-backend.c ##\n @@ refs/packed-backend.c: error:\n 3: 559c04cee99 ! 5: d5669da57e3 refs: alternate reftable ref backend implementation\n @@ Commit message\n refs: alternate reftable ref backend implementation\n \n This introduces the reftable backend as an alternative to the packed\n - backend. This simplifies the code, because we now longer have to worry\n - about:\n + backend.\n +\n + This is an alternative to the approach which was explored in\n + https://github.com/git/git/pull/1215/. This alternative is simpler,\n + because we now longer have to worry about:\n \n - pseudorefs and the HEAD ref\n - worktrees\n - commands that blur together files and references (cherry-pick, rebase)\n \n This deviates from the spec that in\n - Documentation/technical/reftable.txt. It might be possible to update\n - the code such that all writes by default go to reftable directly. Then\n - the result would be compatible with an implementation that writes only\n - reftable (the reftable lock would still prevent races) relative to an\n - implementation that disregards loose refs. Or, JGit could be adapted\n - to follow this implementation.\n + Documentation/technical/reftable.txt. It might be possible to update the\n + code such that all writes by default go to reftable directly. Then the\n + result would be compatible with an implementation that writes only\n + reftable (the reftable lock would still prevent races), but something\n + must be done about the HEAD ref (which JGit keeps in reftable too.)\n + Alternatively, JGit could be adapted to follow this implementation:\n + despite the code being there for 4 years now, I haven't heard of anyone\n + using it in production (exactly because CGit does not support it).\n \n For this incremental path, the reftable format is arguably more\n complex than necessary, as\n \n - packed-refs doesn't support symrefs\n - - reflogs aren't moved into reftable/\n + - reflogs aren't moved into reftable\n \n on the other hand, the code is already there, and it's well-structured\n and well-tested.\n \n - This implementation is a prototype. To test, you need to do `export\n - GIT_TEST_REFTABLE=1`. Doing so creates a handful of errors in the\n - test-suite, most seemingly related to the new behavior of pack-refs\n - (which creates a reftable/ dir and not a packed-refs file.), but it\n - seems overseeable.\n + refs/reftable-backend.c was created by cannibalizing the first version\n + of reftable support (github PR 1215 as mentioned above). It supports\n + reflogs and symrefs, even though those features are never exercised.\n +\n + This implementation is a prototype, for the following reasons:\n +\n + - no considerations of backward compatibility and configuring an\n + extension\n + - no support for converting between packed-refs and reftable\n + - no documentation\n + - test failures when setting GIT_TEST_REFTABLE=1.\n \n Signed-off-by: Han-Wen Nienhuys \n \n @@ contrib/workdir/git-new-workdir: trap cleanup $siglist\n \tcase $x in\n \n ## refs/files-backend.c ##\n +@@\n + #include \"../git-compat-util.h\"\n ++#include \"../abspath.h\"\n + #include \"../config.h\"\n + #include \"../copy.h\"\n + #include \"../environment.h\"\n @@\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n @@ refs/files-backend.c\n #include \"../iterator.h\"\n #include \"../dir-iterator.h\"\n @@ refs/files-backend.c: static struct ref_store *files_ref_store_create(struct repository *repo,\n + \tstruct files_ref_store *refs = xcalloc(1, sizeof(*refs));\n + \tstruct ref_store *ref_store = (struct ref_store *)refs;\n + \tstruct strbuf sb = STRBUF_INIT;\n ++\tint has_reftable, has_packed;\n + \n + \tbase_ref_store_init(ref_store, repo, gitdir, &refs_be_files);\n \trefs->store_flags = flags;\n \tget_common_dir_noenv(&sb, gitdir);\n \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n -+\n -+\t/* TODO: should look at the repo to decide whether to use packed-refs or\n -+\t * reftable. */\n - \trefs->packed_ref_store =\n +-\trefs->packed_ref_store =\n -\t\tpacked_ref_store_create(repo, refs->gitcommondir, flags);\n -+\t\tgit_env_bool(\"GIT_TEST_REFTABLE\", 0) \n ++\n ++\tstrbuf_addf(&sb, \"%s/reftable\", refs->gitcommondir);\n ++\thas_reftable = is_directory(sb.buf);\n ++\tstrbuf_reset(&sb);\n ++\tstrbuf_addf(&sb, \"%s/packed-refs\", refs->gitcommondir);\n ++\thas_packed = file_exists(sb.buf);\n ++\n ++\tif (!has_packed && !has_reftable)\n ++\t\thas_reftable = git_env_bool(\"GIT_TEST_REFTABLE\", 0);\n ++\n ++\trefs->packed_ref_store = has_reftable\n +\t\t? git_reftable_ref_store_create(repo, refs->gitcommondir, flags)\n +\t\t: packed_ref_store_create(repo, refs->gitcommondir, flags);\n \n \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\n \tchdir_notify_reparent(\"files-backend $GIT_COMMONDIR\",\n \n - ## refs/packed-backend.c ##\n -@@ refs/packed-backend.c: static int write_packed_entry(FILE *fh, const char *refname,\n - \treturn 0;\n - }\n - \n --int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n -+static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n - {\n - \tstruct packed_ref_store *refs =\n - \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n -@@ refs/packed-backend.c: int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n - \treturn 0;\n - }\n - \n --void packed_refs_unlock(struct ref_store *ref_store)\n -+static void packed_refs_unlock(struct ref_store *ref_store)\n - {\n - \tstruct packed_ref_store *refs = packed_downcast(\n - \t\t\tref_store,\n -@@ refs/packed-backend.c: void packed_refs_unlock(struct ref_store *ref_store)\n - \trollback_lock_file(&refs->lock);\n - }\n - \n --int packed_refs_is_locked(struct ref_store *ref_store)\n --{\n --\tstruct packed_ref_store *refs = packed_downcast(\n --\t\t\tref_store,\n --\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n --\t\t\t\"packed_refs_is_locked\");\n --\n --\treturn is_lock_file_locked(&refs->lock);\n --}\n --\n - /*\n - * The packed-refs header line that we write out. Perhaps other traits\n - * will be added later.\n -\n ## refs/refs-internal.h ##\n @@ refs/refs-internal.h: struct ref_storage_be {\n };\n @@ refs/reftable-backend.c (new)\n +#include \"../worktree.h\"\n +#include \"refs-internal.h\"\n +#include \"reftable-backend.h\"\n ++#include \"../repository.h\"\n +\n +extern struct ref_storage_be refs_be_reftable;\n +\n @@ refs/reftable-backend.c (new)\n +\treturn &ri->base;\n +}\n +\n -+static int fixup_symrefs(struct ref_store *ref_store,\n -+\t\t\t struct ref_transaction *transaction)\n -+{\n -+\tstruct strbuf referent = STRBUF_INIT;\n -+\tint i = 0;\n -+\tint err = 0;\n -+\n -+\tfor (i = 0; i < transaction->nr; i++) {\n -+\t\tstruct ref_update *update = transaction->updates[i];\n -+\t\tstruct object_id old_oid;\n -+\t\tint failure_errno;\n -+\n -+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n -+\t\t\t\t\t\t&old_oid, &referent,\n -+\t\t\t\t\t\t/* mutate input, like\n -+\t\t\t\t\t\t files-backend.c */\n -+\t\t\t\t\t\t&update->type, &failure_errno);\n -+\t\tif (err < 0 && failure_errno == ENOENT &&\n -+\t\t is_null_oid(&update->old_oid)) {\n -+\t\t\terr = 0;\n -+\t\t}\n -+\t\tif (err < 0)\n -+\t\t\tgoto done;\n -+\n -+\t\tif (!(update->type & REF_ISSYMREF))\n -+\t\t\tcontinue;\n -+\n -+\t\tif (update->flags & REF_NO_DEREF) {\n -+\t\t\t/* what should happen here? See files-backend.c\n -+\t\t\t * lock_ref_for_update. */\n -+\t\t} else {\n -+\t\t\t/*\n -+\t\t\t If we are updating a symref (eg. HEAD), we should also\n -+\t\t\t update the branch that the symref points to.\n -+\n -+\t\t\t This is generic functionality, and would be better\n -+\t\t\t done in refs.c, but the current implementation is\n -+\t\t\t intertwined with the locking in files-backend.c.\n -+\t\t\t*/\n -+\t\t\tint new_flags = update->flags;\n -+\t\t\tstruct ref_update *new_update = NULL;\n -+\n -+\t\t\t/* if this is an update for HEAD, should also record a\n -+\t\t\t log entry for HEAD? See files-backend.c,\n -+\t\t\t split_head_update()\n -+\t\t\t*/\n -+\t\t\tnew_update = ref_transaction_add_update(\n -+\t\t\t\ttransaction, referent.buf, new_flags,\n -+\t\t\t\t&update->new_oid, &update->old_oid,\n -+\t\t\t\tupdate->msg);\n -+\t\t\tnew_update->parent_update = update;\n -+\n -+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n -+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n -+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n -+\t\t}\n -+\t}\n -+\n -+done:\n -+\tassert(err != REFTABLE_API_ERROR);\n -+\tstrbuf_release(&referent);\n -+\treturn err;\n -+}\n -+\n -+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n -+\t\t\t\t\t struct ref_transaction *transaction,\n -+\t\t\t\t\t struct strbuf *errbuf)\n ++static struct ref_transaction *git_reftable_transaction_begin(struct ref_store *ref_store,\n ++\t\t\t\t\t\t\t struct strbuf *errbuf)\n +{\n +\tstruct git_reftable_ref_store *refs =\n +\t\t(struct git_reftable_ref_store *)ref_store;\n +\tstruct reftable_addition *add = NULL;\n -+\tstruct reftable_stack *stack = stack_for(\n -+\t\trefs,\n -+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n -+\tint i;\n ++\tstruct ref_transaction *transaction = NULL;\n ++\tstruct reftable_stack *stack = refs->main_stack;\n +\n +\tint err = refs->err;\n +\tif (err < 0) {\n @@ refs/reftable-backend.c (new)\n +\n +\terr = reftable_stack_new_addition(&add, stack);\n +\tif (err) {\n ++\t\tstrbuf_addf(errbuf, \"reftable: transaction begin: %s\",\n ++\t\t\t reftable_error_str(err));\n +\t\tgoto done;\n +\t}\n +\n -+\tfor (i = 0; i < transaction->nr; i++) {\n -+\t\tstruct ref_update *u = transaction->updates[i];\n -+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n -+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n -+\t\t !(u->flags & REF_LOG_ONLY)) {\n -+\t\t\tstruct object *o =\n -+\t\t\t\tparse_object(refs->base.repo, &u->new_oid);\n -+\t\t\tif (!o) {\n -+\t\t\t\tstrbuf_addf(\n -+\t\t\t\t\terrbuf,\n -+\t\t\t\t\t\"trying to write ref '%s' with nonexistent object %s\",\n -+\t\t\t\t\tu->refname, oid_to_hex(&u->new_oid));\n -+\t\t\t\terr = -1;\n -+\t\t\t\tgoto done;\n -+\t\t\t}\n -+\t\t}\n -+\t}\n -+\n -+\terr = fixup_symrefs(ref_store, transaction);\n -+\tif (err) {\n -+\t\tgoto done;\n -+\t}\n -+\n ++\tCALLOC_ARRAY(transaction, 1);\n +\ttransaction->backend_data = add;\n -+\ttransaction->state = REF_TRANSACTION_PREPARED;\n -+\n +done:\n -+\tassert(err != REFTABLE_API_ERROR);\n -+\tif (err < 0) {\n -+\t\tif (add) {\n -+\t\t\treftable_addition_destroy(add);\n -+\t\t\tadd = NULL;\n -+\t\t}\n -+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n -+\t\tif (!errbuf->len)\n -+\t\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n -+\t\t\t\t reftable_error_str(err));\n -+\t}\n ++\treturn transaction;\n ++}\n +\n -+\treturn err;\n ++static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n ++\t\t\t\t\t struct ref_transaction *transaction,\n ++\t\t\t\t\t struct strbuf *errbuf)\n ++{\n ++\ttransaction->state = REF_TRANSACTION_PREPARED;\n ++\treturn 0;\n +}\n +\n +static int git_reftable_transaction_abort(struct ref_store *ref_store,\n @@ refs/reftable-backend.c (new)\n +\t\t\t\t\t struct ref_transaction *transaction,\n +\t\t\t\t\t struct strbuf *errmsg)\n +{\n ++\tstruct git_reftable_ref_store *refs =\n ++\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n +\tstruct reftable_addition *add =\n +\t\t(struct reftable_addition *)transaction->backend_data;\n +\tint err = 0;\n @@ refs/reftable-backend.c (new)\n +\n +\terr = reftable_addition_commit(add);\n +\n ++\tif (!err)\n ++\t\terr = reftable_stack_auto_compact(refs->main_stack);\n +done:\n +\tassert(err != REFTABLE_API_ERROR);\n +\treftable_addition_destroy(add);\n @@ refs/reftable-backend.c (new)\n +\t.name = \"reftable\",\n +\t.init = git_reftable_ref_store_create,\n +\t.init_db = git_reftable_init_db,\n ++\t.transaction_begin = git_reftable_transaction_begin,\n +\t.transaction_prepare = git_reftable_transaction_prepare,\n +\t.transaction_finish = git_reftable_transaction_finish,\n +\t.transaction_abort = git_reftable_transaction_abort,\n -: ----------- > 6: 2cf743031ab refs: always try to do packed transactions for reftable\n\n-- \ngitgitgadget\n"},{"id":"482058","messageId":"8dedb23eb69493c272dcaaffd98b15a44546c6f9.1695214968.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] refs: push lock management into packed backend","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:45Z","receivedAt":"2023-09-20T13:03:06Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTake packed-refs.lock in the transaction_begin of the packed backend.\n\nThis decouples the files backend from the packed backend, allowing the\nlatter to be swapped out by another ref backend.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs.c | 5 ++-\n refs/files-backend.c | 78 +++++++++++++++----------------------------\n refs/packed-backend.c | 55 ++++++++++++------------------\n refs/packed-backend.h | 10 ------\n 4 files changed, 50 insertions(+), 98 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f7ba3e56284..0611197dc6e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2320,10 +2320,9 @@ int ref_transaction_abort(struct ref_transaction *transaction,\n \tint ret = 0;\n \n \tswitch (transaction->state) {\n-\tcase REF_TRANSACTION_OPEN:\n-\t\t/* No need to abort explicitly. */\n-\t\tbreak;\n \tcase REF_TRANSACTION_PREPARED:\n+\tcase REF_TRANSACTION_OPEN:\n+\t\t/* an OPEN transactions may have a lock. */\n \t\tret = refs->be->transaction_abort(refs, transaction, err);\n \t\tbreak;\n \tcase REF_TRANSACTION_CLOSED:\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d83eea8b408..43c1882be3c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1219,10 +1219,10 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tstruct ref_transaction *transaction;\n \n \ttransaction = ref_store_transaction_begin(refs->packed_ref_store, &err);\n-\tif (!transaction)\n+\tif (!transaction) {\n+\t\tdie(\"could not start transaction: %s\", err.buf);\n \t\treturn -1;\n-\n-\tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n+\t}\n \n \titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n \t\t\t\t\tthe_repository, 0);\n@@ -1262,8 +1262,6 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tref_transaction_free(transaction);\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \tprune_refs(refs, &refs_to_prune);\n \tstrbuf_release(&err);\n \treturn 0;\n@@ -1280,16 +1278,10 @@ static int files_delete_refs(struct ref_store *ref_store, const char *msg,\n \tif (!refnames->nr)\n \t\treturn 0;\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, &err))\n-\t\tgoto error;\n-\n \tif (refs_delete_refs(refs->packed_ref_store, msg, refnames, flags)) {\n-\t\tpacked_refs_unlock(refs->packed_ref_store);\n \t\tgoto error;\n \t}\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \tfor (i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \n@@ -2640,7 +2632,7 @@ out:\n \n struct files_transaction_backend_data {\n \tstruct ref_transaction *packed_transaction;\n-\tint packed_refs_locked;\n+\tint packed_transaction_needed;\n };\n \n /*\n@@ -2672,10 +2664,8 @@ static void files_transaction_cleanup(struct files_ref_store *refs,\n \t\t\tstrbuf_release(&err);\n \t\t}\n \n-\t\tif (backend_data->packed_refs_locked)\n-\t\t\tpacked_refs_unlock(refs->packed_ref_store);\n-\n \t\tfree(backend_data);\n+\t\ttransaction->backend_data = NULL;\n \t}\n \n \ttransaction->state = REF_TRANSACTION_CLOSED;\n@@ -2811,14 +2801,9 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t}\n \n \tif (packed_transaction) {\n-\t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n-\t\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t\tbackend_data->packed_refs_locked = 1;\n-\n-\t\tif (is_packed_transaction_needed(refs->packed_ref_store,\n-\t\t\t\t\t\t packed_transaction)) {\n+\t\tbackend_data->packed_transaction_needed = is_packed_transaction_needed(refs->packed_ref_store,\n+\t\t\t\t\t\t\t\t\t\t packed_transaction);\n+\t\tif (backend_data->packed_transaction_needed) {\n \t\t\tret = ref_transaction_prepare(packed_transaction, err);\n \t\t\t/*\n \t\t\t * A failure during the prepare step will abort\n@@ -2830,22 +2815,6 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t\tref_transaction_free(packed_transaction);\n \t\t\t\tbackend_data->packed_transaction = NULL;\n \t\t\t}\n-\t\t} else {\n-\t\t\t/*\n-\t\t\t * We can skip rewriting the `packed-refs`\n-\t\t\t * file. But we do need to leave it locked, so\n-\t\t\t * that somebody else doesn't pack a reference\n-\t\t\t * that we are trying to delete.\n-\t\t\t *\n-\t\t\t * We need to disconnect our transaction from\n-\t\t\t * backend_data, since the abort (whether successful or\n-\t\t\t * not) will free it.\n-\t\t\t */\n-\t\t\tbackend_data->packed_transaction = NULL;\n-\t\t\tif (ref_transaction_abort(packed_transaction, err)) {\n-\t\t\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n \t\t}\n \t}\n \n@@ -2947,13 +2916,24 @@ static int files_transaction_finish(struct ref_store *ref_store,\n \t * First delete any packed versions of the references, while\n \t * retaining the packed-refs lock:\n \t */\n-\tif (packed_transaction) {\n-\t\tret = ref_transaction_commit(packed_transaction, err);\n-\t\tref_transaction_free(packed_transaction);\n-\t\tpacked_transaction = NULL;\n-\t\tbackend_data->packed_transaction = NULL;\n-\t\tif (ret)\n-\t\t\tgoto cleanup;\n+\tif (backend_data->packed_transaction) {\n+\t\tif (backend_data->packed_transaction_needed) {\n+\t\t\tret = ref_transaction_commit(packed_transaction, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto cleanup;\n+\n+\t\t\tref_transaction_free(packed_transaction);\n+\t\t\tpacked_transaction = NULL;\n+\t\t\tbackend_data->packed_transaction = NULL;\n+\t\t} else {\n+\t\t\tret = ref_transaction_abort(packed_transaction, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto cleanup;\n+\n+\t\t\t/* transaction_commit doesn't free the data, but transaction_abort does. Go figure. */\n+\t\t\tpacked_transaction = NULL;\n+\t\t\tbackend_data->packed_transaction = NULL;\n+\t\t}\n \t}\n \n \t/* Now delete the loose versions of the references: */\n@@ -3094,16 +3074,10 @@ static int files_initial_transaction_commit(struct ref_store *ref_store,\n \t\t\t\t\t NULL);\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n-\t\tret = TRANSACTION_GENERIC_ERROR;\n-\t\tgoto cleanup;\n-\t}\n-\n \tif (initial_ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t}\n \n-\tpacked_refs_unlock(refs->packed_ref_store);\n cleanup:\n \tif (packed_transaction)\n \t\tref_transaction_free(packed_transaction);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 4f5b1025605..f8b27f1afe5 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1153,7 +1153,7 @@ static int write_packed_entry(FILE *fh, const char *refname,\n \treturn 0;\n }\n \n-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n+static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n {\n \tstruct packed_ref_store *refs =\n \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n@@ -1212,7 +1212,7 @@ int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n \treturn 0;\n }\n \n-void packed_refs_unlock(struct ref_store *ref_store)\n+static void packed_refs_unlock(struct ref_store *ref_store)\n {\n \tstruct packed_ref_store *refs = packed_downcast(\n \t\t\tref_store,\n@@ -1224,16 +1224,6 @@ void packed_refs_unlock(struct ref_store *ref_store)\n \trollback_lock_file(&refs->lock);\n }\n \n-int packed_refs_is_locked(struct ref_store *ref_store)\n-{\n-\tstruct packed_ref_store *refs = packed_downcast(\n-\t\t\tref_store,\n-\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n-\t\t\t\"packed_refs_is_locked\");\n-\n-\treturn is_lock_file_locked(&refs->lock);\n-}\n-\n /*\n * The packed-refs header line that we write out. Perhaps other traits\n * will be added later.\n@@ -1552,8 +1542,6 @@ int is_packed_transaction_needed(struct ref_store *ref_store,\n \n struct packed_transaction_backend_data {\n \t/* True iff the transaction owns the packed-refs lock. */\n-\tint own_lock;\n-\n \tstruct string_list updates;\n };\n \n@@ -1568,9 +1556,8 @@ static void packed_transaction_cleanup(struct packed_ref_store *refs,\n \t\tif (is_tempfile_active(refs->tempfile))\n \t\t\tdelete_tempfile(&refs->tempfile);\n \n-\t\tif (data->own_lock && is_lock_file_locked(&refs->lock)) {\n+\t\tif (is_lock_file_locked(&refs->lock)) {\n \t\t\tpacked_refs_unlock(&refs->base);\n-\t\t\tdata->own_lock = 0;\n \t\t}\n \n \t\tfree(data);\n@@ -1580,12 +1567,25 @@ static void packed_transaction_cleanup(struct packed_ref_store *refs,\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n }\n \n-\n static struct ref_transaction *packed_transaction_begin(struct ref_store *ref_store,\n-\t\t\t\t\t\t\tstruct strbuf *err) {\n-\tstruct ref_transaction *tr;\n-\tCALLOC_ARRAY(tr, 1);\n-\treturn tr;\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct packed_ref_store *refs = packed_downcast(\n+\t\t\tref_store,\n+\t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n+\t\t\t\"ref_transaction_begin\");\n+\tstruct packed_transaction_backend_data *data;\n+\tstruct ref_transaction *transaction;\n+\n+\tif (!is_lock_file_locked(&refs->lock)) {\n+\t\tif (packed_refs_lock(ref_store, 0, err))\n+\t\t\treturn NULL;\n+\t}\n+\tCALLOC_ARRAY(transaction, 1);\n+\tCALLOC_ARRAY(data, 1);\n+\tstring_list_init_nodup(&data->updates);\n+\ttransaction->backend_data = data;\n+\treturn transaction;\n }\n \n static int packed_transaction_prepare(struct ref_store *ref_store,\n@@ -1596,7 +1596,7 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \t\t\tref_store,\n \t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n \t\t\t\"ref_transaction_prepare\");\n-\tstruct packed_transaction_backend_data *data;\n+\tstruct packed_transaction_backend_data *data = transaction->backend_data;\n \tsize_t i;\n \tint ret = TRANSACTION_GENERIC_ERROR;\n \n@@ -1609,11 +1609,6 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \t * do so itself.\n \t */\n \n-\tCALLOC_ARRAY(data, 1);\n-\tstring_list_init_nodup(&data->updates);\n-\n-\ttransaction->backend_data = data;\n-\n \t/*\n \t * Stick the updates in a string list by refname so that we\n \t * can sort them:\n@@ -1631,12 +1626,6 @@ static int packed_transaction_prepare(struct ref_store *ref_store,\n \tif (ref_update_reject_duplicates(&data->updates, err))\n \t\tgoto failure;\n \n-\tif (!is_lock_file_locked(&refs->lock)) {\n-\t\tif (packed_refs_lock(ref_store, 0, err))\n-\t\t\tgoto failure;\n-\t\tdata->own_lock = 1;\n-\t}\n-\n \tif (write_with_updates(refs, &data->updates, err))\n \t\tgoto failure;\n \ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex 9dd8a344c34..ade3c8a5ac4 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -17,16 +17,6 @@ struct ref_store *packed_ref_store_create(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags);\n \n-/*\n- * Lock the packed-refs file for writing. Flags is passed to\n- * hold_lock_file_for_update(). Return 0 on success. On errors, write\n- * an error message to `err` and return a nonzero value.\n- */\n-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err);\n-\n-void packed_refs_unlock(struct ref_store *ref_store);\n-int packed_refs_is_locked(struct ref_store *ref_store);\n-\n /*\n * Return true if `transaction` really needs to be carried out against\n * the specified packed_ref_store, or false if it can be skipped\n-- \ngitgitgadget\n\n"},{"id":"482059","messageId":"0b8919b05c43667df1a2163719ec979a07bb9588.1695214968.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] refs: move is_packed_transaction_needed out of packed-backend.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:46Z","receivedAt":"2023-09-20T13:03:09Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nIt is no longer specific to the packed backend.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/files-backend.c | 94 ++++++++++++++++++++++++++++++++++++++++++\n refs/packed-backend.c | 95 -------------------------------------------\n refs/packed-backend.h | 9 ----\n 3 files changed, 94 insertions(+), 104 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 43c1882be3c..5d288bf38bb 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2630,6 +2630,100 @@ out:\n \treturn ret;\n }\n \n+/*\n+ * Return true if `transaction` really needs to be carried out against\n+ * the specified packed_ref_store, or false if it can be skipped\n+ * (i.e., because it is an obvious NOOP). `ref_store` must be locked\n+ * before calling this function.\n+ */\n+static int is_packed_transaction_needed(struct ref_store *ref_store,\n+\t\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * We're only going to bother returning false for the common,\n+\t * trivial case that references are only being deleted, their\n+\t * old values are not being checked, and the old `packed-refs`\n+\t * file doesn't contain any of those reference(s). This gives\n+\t * false positives for some other cases that could\n+\t * theoretically be optimized away:\n+\t *\n+\t * 1. It could be that the old value is being verified without\n+\t * setting a new value. In this case, we could verify the\n+\t * old value here and skip the update if it agrees. If it\n+\t * disagrees, we could either let the update go through\n+\t * (the actual commit would re-detect and report the\n+\t * problem), or come up with a way of reporting such an\n+\t * error to *our* caller.\n+\t *\n+\t * 2. It could be that a new value is being set, but that it\n+\t * is identical to the current packed value of the\n+\t * reference.\n+\t *\n+\t * Neither of these cases will come up in the current code,\n+\t * because the only caller of this function passes to it a\n+\t * transaction that only includes `delete` updates with no\n+\t * `old_id`. Even if that ever changes, false positives only\n+\t * cause an optimization to be missed; they do not affect\n+\t * correctness.\n+\t */\n+\n+\t/*\n+\t * Start with the cheap checks that don't require old\n+\t * reference values to be read:\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\n+\t\tif (update->flags & REF_HAVE_OLD)\n+\t\t\t/* Have to check the old value -> needed. */\n+\t\t\treturn 1;\n+\n+\t\tif ((update->flags & REF_HAVE_NEW) && !is_null_oid(&update->new_oid))\n+\t\t\t/* Have to set a new value -> needed. */\n+\t\t\treturn 1;\n+\t}\n+\n+\t/*\n+\t * The transaction isn't checking any old values nor is it\n+\t * setting any nonzero new values, so it still might be able\n+\t * to be skipped. Now do the more expensive check: the update\n+\t * is needed if any of the updates is a delete, and the old\n+\t * `packed-refs` file contains a value for that reference.\n+\t */\n+\tret = 0;\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tint failure_errno;\n+\t\tunsigned int type;\n+\t\tstruct object_id oid;\n+\n+\t\tif (!(update->flags & REF_HAVE_NEW))\n+\t\t\t/*\n+\t\t\t * This reference isn't being deleted -> not\n+\t\t\t * needed.\n+\t\t\t */\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_read_raw_ref(ref_store, update->refname, &oid,\n+\t\t\t\t &referent, &type, &failure_errno) ||\n+\t\t failure_errno != ENOENT) {\n+\t\t\t/*\n+\t\t\t * We have to actually delete that reference\n+\t\t\t * -> this transaction is needed.\n+\t\t\t */\n+\t\t\tret = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n struct files_transaction_backend_data {\n \tstruct ref_transaction *packed_transaction;\n \tint packed_transaction_needed;\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f8b27f1afe5..2017d117c13 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1445,101 +1445,6 @@ error:\n \treturn -1;\n }\n \n-int is_packed_transaction_needed(struct ref_store *ref_store,\n-\t\t\t\t struct ref_transaction *transaction)\n-{\n-\tstruct packed_ref_store *refs = packed_downcast(\n-\t\t\tref_store,\n-\t\t\tREF_STORE_READ,\n-\t\t\t\"is_packed_transaction_needed\");\n-\tstruct strbuf referent = STRBUF_INIT;\n-\tsize_t i;\n-\tint ret;\n-\n-\tif (!is_lock_file_locked(&refs->lock))\n-\t\tBUG(\"is_packed_transaction_needed() called while unlocked\");\n-\n-\t/*\n-\t * We're only going to bother returning false for the common,\n-\t * trivial case that references are only being deleted, their\n-\t * old values are not being checked, and the old `packed-refs`\n-\t * file doesn't contain any of those reference(s). This gives\n-\t * false positives for some other cases that could\n-\t * theoretically be optimized away:\n-\t *\n-\t * 1. It could be that the old value is being verified without\n-\t * setting a new value. In this case, we could verify the\n-\t * old value here and skip the update if it agrees. If it\n-\t * disagrees, we could either let the update go through\n-\t * (the actual commit would re-detect and report the\n-\t * problem), or come up with a way of reporting such an\n-\t * error to *our* caller.\n-\t *\n-\t * 2. It could be that a new value is being set, but that it\n-\t * is identical to the current packed value of the\n-\t * reference.\n-\t *\n-\t * Neither of these cases will come up in the current code,\n-\t * because the only caller of this function passes to it a\n-\t * transaction that only includes `delete` updates with no\n-\t * `old_id`. Even if that ever changes, false positives only\n-\t * cause an optimization to be missed; they do not affect\n-\t * correctness.\n-\t */\n-\n-\t/*\n-\t * Start with the cheap checks that don't require old\n-\t * reference values to be read:\n-\t */\n-\tfor (i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n-\n-\t\tif (update->flags & REF_HAVE_OLD)\n-\t\t\t/* Have to check the old value -> needed. */\n-\t\t\treturn 1;\n-\n-\t\tif ((update->flags & REF_HAVE_NEW) && !is_null_oid(&update->new_oid))\n-\t\t\t/* Have to set a new value -> needed. */\n-\t\t\treturn 1;\n-\t}\n-\n-\t/*\n-\t * The transaction isn't checking any old values nor is it\n-\t * setting any nonzero new values, so it still might be able\n-\t * to be skipped. Now do the more expensive check: the update\n-\t * is needed if any of the updates is a delete, and the old\n-\t * `packed-refs` file contains a value for that reference.\n-\t */\n-\tret = 0;\n-\tfor (i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n-\t\tint failure_errno;\n-\t\tunsigned int type;\n-\t\tstruct object_id oid;\n-\n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\t/*\n-\t\t\t * This reference isn't being deleted -> not\n-\t\t\t * needed.\n-\t\t\t */\n-\t\t\tcontinue;\n-\n-\t\tif (!refs_read_raw_ref(ref_store, update->refname, &oid,\n-\t\t\t\t &referent, &type, &failure_errno) ||\n-\t\t failure_errno != ENOENT) {\n-\t\t\t/*\n-\t\t\t * We have to actually delete that reference\n-\t\t\t * -> this transaction is needed.\n-\t\t\t */\n-\t\t\tret = 1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\tstrbuf_release(&referent);\n-\treturn ret;\n-}\n-\n struct packed_transaction_backend_data {\n \t/* True iff the transaction owns the packed-refs lock. */\n \tstruct string_list updates;\ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex ade3c8a5ac4..51a3b6a332a 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -17,13 +17,4 @@ struct ref_store *packed_ref_store_create(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags);\n \n-/*\n- * Return true if `transaction` really needs to be carried out against\n- * the specified packed_ref_store, or false if it can be skipped\n- * (i.e., because it is an obvious NOOP). `ref_store` must be locked\n- * before calling this function.\n- */\n-int is_packed_transaction_needed(struct ref_store *ref_store,\n-\t\t\t\t struct ref_transaction *transaction);\n-\n #endif /* REFS_PACKED_BACKEND_H */\n-- \ngitgitgadget\n\n"},{"id":"482060","messageId":"2cf743031abbbd3de94d98dc883dfc01f327aadc.1695214969.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] refs: always try to do packed transactions for reftable","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:48Z","receivedAt":"2023-09-20T13:03:17Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nReftable updates are cheap (compared to packed-refs updates), so do them\nalways. This also helps exercise this code more in tests.\n\nThis should commit should not be merged currently. It fails 55 tests\nwhen running with GIT_TEST_REFTABLE.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/files-backend.c | 22 +++++++++++++++++-----\n 1 file changed, 17 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 2cd596bbeba..0d61a18b4ba 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -2800,6 +2800,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n \tstruct ref_transaction *packed_transaction = NULL;\n+\tint is_reftable = !strcmp(refs->packed_ref_store->be->name, \"reftable\");\n \n \tassert(err);\n \n@@ -2809,6 +2810,16 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tCALLOC_ARRAY(backend_data, 1);\n \ttransaction->backend_data = backend_data;\n \n+\tif (is_reftable) {\n+\t\tpacked_transaction = ref_store_transaction_begin(\n+\t\t\trefs->packed_ref_store, err);\n+\t\tif (!packed_transaction) {\n+\t\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t\tbackend_data->packed_transaction = packed_transaction;\n+\t}\n+\n \t/*\n \t * Fail if a refname appears more than once in the\n \t * transaction. (If we end up splitting up any updates using\n@@ -2881,9 +2892,9 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n-\t\tif (update->flags & REF_DELETING &&\n-\t\t !(update->flags & REF_LOG_ONLY) &&\n-\t\t !(update->flags & REF_IS_PRUNING)) {\n+\t\tif (is_reftable || (update->flags & REF_DELETING &&\n+\t\t\t\t !(update->flags & REF_LOG_ONLY) &&\n+\t\t\t\t !(update->flags & REF_IS_PRUNING))) {\n \t\t\t/*\n \t\t\t * This reference has to be deleted from\n \t\t\t * packed-refs if it exists there.\n@@ -2909,8 +2920,9 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t}\n \n \tif (packed_transaction) {\n-\t\tbackend_data->packed_transaction_needed = is_packed_transaction_needed(refs->packed_ref_store,\n-\t\t\t\t\t\t\t\t\t\t packed_transaction);\n+\t\tbackend_data->packed_transaction_needed = is_reftable ||\n+\t\t\tis_packed_transaction_needed(refs->packed_ref_store,\n+\t\t\t\t\t\t packed_transaction);\n \t\tif (backend_data->packed_transaction_needed) {\n \t\t\tret = ref_transaction_prepare(packed_transaction, err);\n \t\t\t/*\n-- \ngitgitgadget\n"},{"id":"482061","messageId":"d5669da57e33ff2fbd6fad9e68285c33b43061db.1695214968.git.gitgitgadget@gmail.com","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] refs: alternate reftable ref backend implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-09-20T13:02:47Z","receivedAt":"2023-09-20T13:03:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis introduces the reftable backend as an alternative to the packed\nbackend.\n\nThis is an alternative to the approach which was explored in\nhttps://github.com/git/git/pull/1215/. This alternative is simpler,\nbecause we now longer have to worry about:\n\n- pseudorefs and the HEAD ref\n- worktrees\n- commands that blur together files and references (cherry-pick, rebase)\n\nThis deviates from the spec that in\nDocumentation/technical/reftable.txt. It might be possible to update the\ncode such that all writes by default go to reftable directly. Then the\nresult would be compatible with an implementation that writes only\nreftable (the reftable lock would still prevent races), but something\nmust be done about the HEAD ref (which JGit keeps in reftable too.)\nAlternatively, JGit could be adapted to follow this implementation:\ndespite the code being there for 4 years now, I haven't heard of anyone\nusing it in production (exactly because CGit does not support it).\n\nFor this incremental path, the reftable format is arguably more\ncomplex than necessary, as\n\n- packed-refs doesn't support symrefs\n- reflogs aren't moved into reftable\n\non the other hand, the code is already there, and it's well-structured\nand well-tested.\n\nrefs/reftable-backend.c was created by cannibalizing the first version\nof reftable support (github PR 1215 as mentioned above). It supports\nreflogs and symrefs, even though those features are never exercised.\n\nThis implementation is a prototype, for the following reasons:\n\n- no considerations of backward compatibility and configuring an\nextension\n- no support for converting between packed-refs and reftable\n- no documentation\n- test failures when setting GIT_TEST_REFTABLE=1.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n config.mak.uname | 2 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs/files-backend.c | 18 +-\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1658 +++++++++++++++++++++++++++++++\n refs/reftable-backend.h | 8 +\n 7 files changed, 1686 insertions(+), 4 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 refs/reftable-backend.h\n\ndiff --git a/Makefile b/Makefile\nindex 57763093653..272d3f7f1e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1118,6 +1118,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 3bb03f423a0..843829c02fd 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -743,7 +743,7 @@ vcxproj:\n \n \t# Make .vcxproj files and add them\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a5215..989197aace0 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5d288bf38bb..2cd596bbeba 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,4 +1,5 @@\n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../config.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n@@ -9,6 +10,7 @@\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n #include \"packed-backend.h\"\n+#include \"reftable-backend.h\"\n #include \"../ident.h\"\n #include \"../iterator.h\"\n #include \"../dir-iterator.h\"\n@@ -98,13 +100,25 @@ static struct ref_store *files_ref_store_create(struct repository *repo,\n \tstruct files_ref_store *refs = xcalloc(1, sizeof(*refs));\n \tstruct ref_store *ref_store = (struct ref_store *)refs;\n \tstruct strbuf sb = STRBUF_INIT;\n+\tint has_reftable, has_packed;\n \n \tbase_ref_store_init(ref_store, repo, gitdir, &refs_be_files);\n \trefs->store_flags = flags;\n \tget_common_dir_noenv(&sb, gitdir);\n \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n-\trefs->packed_ref_store =\n-\t\tpacked_ref_store_create(repo, refs->gitcommondir, flags);\n+\n+\tstrbuf_addf(&sb, \"%s/reftable\", refs->gitcommondir);\n+\thas_reftable = is_directory(sb.buf);\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"%s/packed-refs\", refs->gitcommondir);\n+\thas_packed = file_exists(sb.buf);\n+\n+\tif (!has_packed && !has_reftable)\n+\t\thas_reftable = git_env_bool(\"GIT_TEST_REFTABLE\", 0);\n+\n+\trefs->packed_ref_store = has_reftable\n+\t\t? git_reftable_ref_store_create(repo, refs->gitcommondir, flags)\n+\t\t: packed_ref_store_create(repo, refs->gitcommondir, flags);\n \n \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\n \tchdir_notify_reparent(\"files-backend $GIT_COMMONDIR\",\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 0a15e8a2ac8..13bd9c79f3c 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -699,6 +699,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..16064fff198\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1658 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../environment.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+#include \"reftable-backend.h\"\n+#include \"../repository.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\n+\tstruct reftable_write_options write_options;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\treturn store->main_stack;\n+}\n+\n+static int should_log(const char *refname)\n+{\n+\treturn log_all_ref_updates != LOG_REFS_NONE &&\n+\t (log_all_ref_updates == LOG_REFS_ALWAYS ||\n+\t\tlog_all_ref_updates == LOG_REFS_UNSET ||\n+\t\tshould_autocreate_reflog(refname));\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *stripped;\n+\tparse_worktree_ref(ref, NULL, NULL, &stripped);\n+\treturn stripped;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type, int *failure_errno);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/* when we write log records, the hashes are owned by a struct\n+\t\t * oid */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+struct ref_store *git_reftable_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n+\t\t\t\t\t\tunsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint shared = get_shared_repository();\n+\tif (shared < 0)\n+\t\tshared = -shared;\n+\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = the_hash_algo->format_id;\n+\tif (shared && (shared & 0600))\n+\t\trefs->write_options.default_permissions = shared;\n+\n+\t/* XXX should this use `path` or `gitdir.buf` ? */\n+\tbase_ref_store_init(ref_store, repo, gitdir, &refs_be_reftable);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\trefs->base.repo = repo;\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err = reftable_new_stack(&refs->main_stack, refs->reftable_dir,\n+\t\t\t\t refs->write_options);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&sb);\n+\n+\t/* TODO something with chdir_notify_reparent() ? */\n+\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ri->ref_store;\n+\n+\twhile (ri->err == 0) {\n+\t\tint signed_flags = 0;\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tri->base.flags = 0;\n+\n+\t\tif (!strcmp(ri->ref.refname, \"HEAD\")) {\n+\t\t\t/*\n+\t\t\t HEAD should not be produced by default.Other\n+\t\t\t pseudorefs (FETCH_HEAD etc.) shouldn't be\n+\t\t\t stored in reftables at all.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(ri->base.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tif (ri->flags & DO_FOR_EACH_INCLUDE_BROKEN &&\n+\t\t check_refname_format(ri->base.refname,\n+\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\t/* This is odd, as REF_BAD_NAME and REF_ISBROKEN are\n+\t\t\t orthogonal, but it's what the spec says and the\n+\t\t\t files-backend does. */\n+\t\t\tri->base.flags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tri->base.flags = REF_ISSYMREF;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, refs->base.repo,\n+\t\t\t\t\t ri->base.oid, ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Arguably, resolving recursively following symlinks should be\n+\t\t * lifted to refs.c because it is shared between reftable and\n+\t\t * the files backend, but it's here now.\n+\t\t */\n+\t\tif (!refs_resolve_ref_unsafe(ri->ref_store, ri->ref.refname,\n+\t\t\t\t\t RESOLVE_REF_READING, &ri->oid,\n+\t\t\t\t\t &signed_flags)) {\n+\t\t\tri->base.flags = signed_flags;\n+\t\t\tif (ri->ref.value_type == REFTABLE_REF_SYMREF &&\n+\t\t\t ri->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (ri->ref.value_type == REFTABLE_REF_SYMREF &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (is_null_oid(&ri->oid)) {\n+\t\t\t\toidclr(&ri->oid);\n+\t\t\t\tri->base.flags |= REF_ISBROKEN;\n+\t\t\t}\n+\t\t}\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tconst char **exclude_patterns_TODO,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static struct ref_transaction *git_reftable_transaction_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct ref_transaction *transaction = NULL;\n+\tstruct reftable_stack *stack = refs->main_stack;\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction begin: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\tgoto done;\n+\t}\n+\n+\tCALLOC_ARRAY(transaction, 1);\n+\ttransaction->backend_data = add;\n+done:\n+\treturn transaction;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\treturn 0;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\n+\t/* XXX. Shouldn't this be handled generically in refs.c? */\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != !resolved) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tint log_count = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[log_count];\n+\t\tstruct object_id old_id = *null_oid();\n+\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tif ((u->flags & REF_FORCE_CREATE_REFLOG) ||\n+\t\t should_log(u->refname))\n+\t\t\tlog_count++;\n+\t\tfill_reftable_log_record(log);\n+\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = xstrdup(u->refname);\n+\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\tlog->value.update.message =\n+\t\t\txstrndup(u->msg, refs->write_options.block_size / 2);\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\t\tlog->value.update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < log_count; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+\tif (!err)\n+\t\terr = reftable_stack_auto_compact(refs->main_stack);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = xstrdup(arg->refnames->items[i].string);\n+\t\tif (!should_log(log.refname)) {\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlog.value.update.message = xstrndup(\n+\t\t\targ->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.value.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.value.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = xstrdup(create->refname);\n+\t\tif (!should_log(log.refname)) {\n+\t\t\treturn err;\n+\t\t}\n+\t\tlog.update_index = ts;\n+\t\tlog.value.update.message =\n+\t\t\txstrndup(create->logmsg,\n+\t\t\t\t create->refs->write_options.block_size / 2);\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL)) {\n+\t\t\tlog.value.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL)) {\n+\t\t\tlog.value.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.value.update.old_hash ||\n+\t\t log.value.update.new_hash) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.value.update.message = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tstruct reftable_ref_record todo[2] = {\n+\t\t{\n+\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t.update_index = ts,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t},\n+\t\told_ref,\n+\t};\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (!reftable_stack_read_ref(arg->stack, arg->newname, &new_ref)) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\ttodo[1].update_index = ts;\n+\ttodo[1].refname = (char *)arg->newname;\n+\n+\terr = reftable_writer_add_refs(writer, todo, 2);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tint firstlog = 0;\n+\t\tint lastlog = 2;\n+\t\tchar *msg = xstrndup(arg->logmsg,\n+\t\t\t\t arg->refs->write_options.block_size / 2);\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = xstrdup(arg->oldname);\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].value.update.message = msg;\n+\t\ttodo[0].value.update.old_hash = val1;\n+\t\ttodo[0].value.update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = xstrdup(arg->newname);\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].value.update.old_hash = NULL;\n+\t\ttodo[1].value.update.new_hash = val1;\n+\t\ttodo[1].value.update.message = xstrdup(msg);\n+\n+\t\tif (!should_log(todo[1].refname)) {\n+\t\t\tlastlog--;\n+\t\t}\n+\t\tif (!should_log(todo[0].refname)) {\n+\t\t\tfirstlog++;\n+\t\t}\n+\t\terr = reftable_writer_add_logs(writer, &todo[firstlog],\n+\t\t\t\t\t lastlog - firstlog);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX what should we write into the reflog if we rename a\n+\t\t * symref? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX this copies the entire reflog history. Is this the right\n+\t * semantics? should clear out existing reflog entries for oldname? */\n+\tif (!should_log(arg->newname))\n+\t\tgoto done;\n+\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tstruct git_reftable_ref_store *refs;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint flags = 0;\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!refs_resolve_ref_unsafe(&ri->refs->base, ri->log.refname,\n+\t\t\t\t\t 0, &ri->oid, &flags)) {\n+\t\t\terror(\"bad ref for %s\", ri->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\tri->base.oid = &ri->oid;\n+\t\tri->base.flags = flags;\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct reftable_stack *stack = refs->main_stack;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(stack);\n+\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\n+\tri->refs = refs;\n+\tif (err < 0) {\n+\t\tfree(ri);\n+\t\t/* XXX how to handle errors in iterator_begin()? */\n+\t\treturn NULL;\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\t/* placeholder for existence. */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfull_committer = fmt_ident(log.value.update.name,\n+\t\t\t\t\t log.value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log.value.update.time, log.value.update.tz_offset,\n+\t\t\t log.value.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->value.update.old_hash);\n+\t\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\t/* placeholder for existence. */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfull_committer = fmt_ident(log->value.update.name,\n+\t\t\t\t\t log->value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t\t log->value.update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *argv)\n+{\n+\tstruct write_reflog_existence_arg *arg =\n+\t\t(struct write_reflog_existence_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tint err = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (err <= 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tlog.refname = (char *)arg->refname;\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\terr = reftable_writer_add_log(writer, &log);\n+\n+\t/* field is not malloced */\n+\tlog.refname = NULL;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn err;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (log.value_type == REFTABLE_LOG_DELETION)\n+\t\t\tcontinue;\n+\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\tint live_records = 0;\n+\tuint64_t max_ts = 0;\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\t\tif (max_ts < arg->records[i].update_index)\n+\t\t\tmax_ts = arg->records[i].update_index;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tif (live_records == 0) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = max_ts + 1,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t/* existence dummy has null new/old oid */\n+\t\t};\n+\t\tint err;\n+\t\tif (log.update_index < ts)\n+\t\t\tlog.update_index = ts;\n+\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_reflog_expire(\n+\tstruct ref_store *ref_store, const char *refname, unsigned int flags,\n+\treflog_expiry_prepare_fn prepare_fn,\n+\treflog_expiry_should_prune_fn should_prune_fn,\n+\treflog_expiry_cleanup_fn cleanup_fn, void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\n+\t On the plus side, because we do the expiration per ref, we can easily\n+\t insert the reflog existence dummies.\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = { NULL };\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tstruct object_id oid = { 0 };\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tif (!reftable_stack_read_ref(stack, refname, &ref_record)) {\n+\t\tuint8_t *hash = reftable_ref_record_val1(&ref_record);\n+\t\tif (hash)\n+\t\t\toidread(&oid, hash);\n+\t}\n+\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].value.update.old_hash);\n+\t\toidread(&noid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash) {\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* future improvement: we could skip writing records that were\n+\t\t * not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type, int *failure_errno)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err == 0 && ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t} else {\n+\t\terr = -1;\n+\t}\n+\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.next = &refs_be_files,\n+\t.name = \"reftable\",\n+\t.init = git_reftable_ref_store_create,\n+\t.init_db = git_reftable_init_db,\n+\t.transaction_begin = git_reftable_transaction_begin,\n+\t.transaction_prepare = git_reftable_transaction_prepare,\n+\t.transaction_finish = git_reftable_transaction_finish,\n+\t.transaction_abort = git_reftable_transaction_abort,\n+\t.initial_transaction_commit = git_reftable_transaction_initial_commit,\n+\n+\t.pack_refs = git_reftable_pack_refs,\n+\t.create_symref = git_reftable_create_symref,\n+\t.delete_refs = git_reftable_delete_refs,\n+\t.rename_ref = git_reftable_rename_ref,\n+\t.copy_ref = git_reftable_copy_ref,\n+\n+\t.iterator_begin = git_reftable_ref_iterator_begin,\n+\t.read_raw_ref = git_reftable_read_raw_ref,\n+\t.read_symbolic_ref = git_reftable_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = git_reftable_reflog_iterator_begin,\n+\t.for_each_reflog_ent = git_reftable_for_each_reflog_ent_oldest_first,\n+\t.for_each_reflog_ent_reverse =\n+\t\tgit_reftable_for_each_reflog_ent_newest_first,\n+\t.reflog_exists = git_reftable_reflog_exists,\n+\t.create_reflog = git_reftable_create_reflog,\n+\t.delete_reflog = git_reftable_delete_reflog,\n+\t.reflog_expire = git_reftable_reflog_expire,\n+};\ndiff --git a/refs/reftable-backend.h b/refs/reftable-backend.h\nnew file mode 100644\nindex 00000000000..bf5fd42229a\n--- /dev/null\n+++ b/refs/reftable-backend.h\n@@ -0,0 +1,8 @@\n+#ifndef REFS_REFTABLE_BACKEND_H\n+#define REFS_REFTABLE_BACKEND_H\n+\n+struct ref_store *git_reftable_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n+\t\t\t\t\t\tunsigned int store_flags);\n+\n+#endif /* REFS_REFTABLE_BACKEND_H */\n-- \ngitgitgadget\n\n"},{"id":"482092","messageId":"ZQwUbFn3WiRHVkT-@tanuki","threadId":"60240","inReplyTo":"pull.1574.v2.git.git.1695214968.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/6] RFC: simple reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-09-21T10:01:16Z","receivedAt":"2023-09-21T17:10:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 20, 2023 at 01:02:42PM +0000, Han-Wen Nienhuys via GitGitGadget wrote:\n> This series comes from a conversation with Patrick Steinhardt at Gitlab, who\n> have an interest in a more scalable ref storage system.\n> \n> I unfortunately don't have business reasons to spend much time on this\n> project anymore, and the original, sweeping migration has too many open\n> questions.\n> \n> I thought of an alternate approach, and I wanted to show it as input to\n> discussions at the contributor summit.\n> \n> I think the first part (everything before \"refs: alternate reftable ref\n> backend implementation\") is a good improvement overall, and could be landed\n> separately without much tweaking.\n> \n> The second part (\"refs: alternate reftable ref backend implementation\" and\n> follow-on) is something open for discussion: the alternative \"packed\n> storage\" is based on reftable, but it could conceivably be a different type\n> of database/file format too. I think it's attractive to use reftable here,\n> because over time more data (symrefs, reflog) could be moved into reftable.\n> \n> I'm offering this series as a RFC. I hope that someone else (Patrick?) can\n> look into getting the bits and pieces of this merged.\n\nI've added this topic to the Contributor's summit such that the project\ncan discuss what to do about the different proposals that exist around\nnext-gen reference backends. If the conclusion is that reftables and\nthis proposal here is still likely to land, then I'm happy to pick it up\nand try to get it upstreamed.\n\nI'll have a look at this RFC early next week.\n\nThanks!\n\nPatrick\n\n> Han-Wen Nienhuys (6):\n> refs: construct transaction using a _begin callback\n> refs: wrap transaction in a debug-specific transaction\n> refs: push lock management into packed backend\n> refs: move is_packed_transaction_needed out of packed-backend.c\n> refs: alternate reftable ref backend implementation\n> refs: always try to do packed transactions for reftable\n> \n> Makefile | 1 +\n> config.mak.uname | 2 +-\n> contrib/workdir/git-new-workdir | 2 +-\n> refs.c | 13 +-\n> refs/debug.c | 87 +-\n> refs/files-backend.c | 216 ++--\n> refs/packed-backend.c | 149 +--\n> refs/packed-backend.h | 19 -\n> refs/refs-internal.h | 5 +\n> refs/reftable-backend.c | 1658 +++++++++++++++++++++++++++++++\n> refs/reftable-backend.h | 8 +\n> 11 files changed, 1929 insertions(+), 231 deletions(-)\n> create mode 100644 refs/reftable-backend.c\n> create mode 100644 refs/reftable-backend.h\n> \n> \n> base-commit: bda494f4043963b9ec9a1ecd4b19b7d1cd9a0518\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1574%2Fhanwen%2Fsimple-reftable-backend-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1574/hanwen/simple-reftable-backend-v2\n> Pull-Request: https://github.com/git/git/pull/1574\n> \n> Range-diff vs v1:\n> \n> -: ----------- > 1: e99f3d20056 refs: construct transaction using a _begin callback\n> -: ----------- > 2: 9a459259330 refs: wrap transaction in a debug-specific transaction\n> 1: dea0fbb139a ! 3: 8dedb23eb69 refs: push lock management into packed backend\n> @@ Metadata\n> ## Commit message ##\n> refs: push lock management into packed backend\n> \n> - Introduces a ref backend method transaction_begin, which for the\n> - packed backend takes packed-refs.lock.\n> + Take packed-refs.lock in the transaction_begin of the packed backend.\n> \n> This decouples the files backend from the packed backend, allowing the\n> latter to be swapped out by another ref backend.\n> @@ Commit message\n> Signed-off-by: Han-Wen Nienhuys \n> \n> ## refs.c ##\n> -@@ refs.c: struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,\n> - \t\t\t\t\t\t struct strbuf *err)\n> - {\n> - \tstruct ref_transaction *tr;\n> -+\tint ret = 0;\n> - \tassert(err);\n> - \n> - \tCALLOC_ARRAY(tr, 1);\n> - \ttr->ref_store = refs;\n> -+\n> -+\tif (refs->be->transaction_begin)\n> -+\t\tret = refs->be->transaction_begin(refs, tr, err);\n> -+\tif (ret) {\n> -+\t\tfree(tr);\n> -+\t\treturn NULL;\n> -+\t}\n> - \treturn tr;\n> - }\n> - \n> -\n> - ## refs/debug.c ##\n> -@@ refs/debug.c: static int debug_init_db(struct ref_store *refs, struct strbuf *err)\n> - \treturn res;\n> - }\n> - \n> -+static int debug_transaction_begin(struct ref_store *refs,\n> -+\t\t\t\t struct ref_transaction *transaction,\n> -+\t\t\t\t struct strbuf *err)\n> -+{\n> -+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)refs;\n> -+\tint res;\n> -+\ttransaction->ref_store = drefs->refs;\n> -+\tres = drefs->refs->be->transaction_begin(drefs->refs, transaction,\n> -+\t\t\t\t\t\t err);\n> -+\ttrace_printf_key(&trace_refs, \"transaction_begin: %d \\\"%s\\\"\\n\", res,\n> -+\t\t\t err->buf);\n> -+\treturn res;\n> -+}\n> -+\n> - static int debug_transaction_prepare(struct ref_store *refs,\n> - \t\t\t\t struct ref_transaction *transaction,\n> - \t\t\t\t struct strbuf *err)\n> -@@ refs/debug.c: struct ref_storage_be refs_be_debug = {\n> - \t * has a function we should also have a wrapper for it here.\n> - \t * Test the output with \"GIT_TRACE_REFS=1\".\n> - \t */\n> -+\t.transaction_begin = debug_transaction_begin,\n> - \t.transaction_prepare = debug_transaction_prepare,\n> - \t.transaction_finish = debug_transaction_finish,\n> - \t.transaction_abort = debug_transaction_abort,\n> +@@ refs.c: int ref_transaction_abort(struct ref_transaction *transaction,\n> + \tint ret = 0;\n> + \n> + \tswitch (transaction->state) {\n> +-\tcase REF_TRANSACTION_OPEN:\n> +-\t\t/* No need to abort explicitly. */\n> +-\t\tbreak;\n> + \tcase REF_TRANSACTION_PREPARED:\n> ++\tcase REF_TRANSACTION_OPEN:\n> ++\t\t/* an OPEN transactions may have a lock. */\n> + \t\tret = refs->be->transaction_abort(refs, transaction, err);\n> + \t\tbreak;\n> + \tcase REF_TRANSACTION_CLOSED:\n> \n> ## refs/files-backend.c ##\n> @@ refs/files-backend.c: static int files_pack_refs(struct ref_store *ref_store,\n> @@ refs/files-backend.c: static int files_transaction_finish(struct ref_store *ref_\n> -\t\tbackend_data->packed_transaction = NULL;\n> -\t\tif (ret)\n> -\t\t\tgoto cleanup;\n> -+\tif (backend_data->packed_transaction) { \n> ++\tif (backend_data->packed_transaction) {\n> +\t\tif (backend_data->packed_transaction_needed) {\n> +\t\t\tret = ref_transaction_commit(packed_transaction, err);\n> +\t\t\tif (ret)\n> +\t\t\t\tgoto cleanup;\n> -+\t\t\t/* TODO: leaks on error path. */\n> ++\n> +\t\t\tref_transaction_free(packed_transaction);\n> +\t\t\tpacked_transaction = NULL;\n> +\t\t\tbackend_data->packed_transaction = NULL;\n> @@ refs/files-backend.c: static int files_initial_transaction_commit(struct ref_sto\n> \t\tref_transaction_free(packed_transaction);\n> \n> ## refs/packed-backend.c ##\n> +@@ refs/packed-backend.c: static int write_packed_entry(FILE *fh, const char *refname,\n> + \treturn 0;\n> + }\n> + \n> +-int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> ++static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> + {\n> + \tstruct packed_ref_store *refs =\n> + \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n> +@@ refs/packed-backend.c: int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> + \treturn 0;\n> + }\n> + \n> +-void packed_refs_unlock(struct ref_store *ref_store)\n> ++static void packed_refs_unlock(struct ref_store *ref_store)\n> + {\n> + \tstruct packed_ref_store *refs = packed_downcast(\n> + \t\t\tref_store,\n> +@@ refs/packed-backend.c: void packed_refs_unlock(struct ref_store *ref_store)\n> + \trollback_lock_file(&refs->lock);\n> + }\n> + \n> +-int packed_refs_is_locked(struct ref_store *ref_store)\n> +-{\n> +-\tstruct packed_ref_store *refs = packed_downcast(\n> +-\t\t\tref_store,\n> +-\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n> +-\t\t\t\"packed_refs_is_locked\");\n> +-\n> +-\treturn is_lock_file_locked(&refs->lock);\n> +-}\n> +-\n> + /*\n> + * The packed-refs header line that we write out. Perhaps other traits\n> + * will be added later.\n> @@ refs/packed-backend.c: int is_packed_transaction_needed(struct ref_store *ref_store,\n> \n> struct packed_transaction_backend_data {\n> @@ refs/packed-backend.c: static void packed_transaction_cleanup(struct packed_ref_\n> \ttransaction->state = REF_TRANSACTION_CLOSED;\n> }\n> \n> -+static int packed_transaction_begin(struct ref_store *ref_store,\n> -+\t\t\t\t struct ref_transaction *transaction,\n> +-\n> + static struct ref_transaction *packed_transaction_begin(struct ref_store *ref_store,\n> +-\t\t\t\t\t\t\tstruct strbuf *err) {\n> +-\tstruct ref_transaction *tr;\n> +-\tCALLOC_ARRAY(tr, 1);\n> +-\treturn tr;\n> +\t\t\t\t struct strbuf *err)\n> +{\n> +\tstruct packed_ref_store *refs = packed_downcast(\n> @@ refs/packed-backend.c: static void packed_transaction_cleanup(struct packed_ref_\n> +\t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n> +\t\t\t\"ref_transaction_begin\");\n> +\tstruct packed_transaction_backend_data *data;\n> -+\n> -+\tCALLOC_ARRAY(data, 1);\n> -+\tstring_list_init_nodup(&data->updates);\n> ++\tstruct ref_transaction *transaction;\n> +\n> +\tif (!is_lock_file_locked(&refs->lock)) {\n> +\t\tif (packed_refs_lock(ref_store, 0, err))\n> -+\t\t\t/* todo: leaking data */\n> -+\t\t\treturn -1;\n> ++\t\t\treturn NULL;\n> +\t}\n> ++\tCALLOC_ARRAY(transaction, 1);\n> ++\tCALLOC_ARRAY(data, 1);\n> ++\tstring_list_init_nodup(&data->updates);\n> +\ttransaction->backend_data = data;\n> -+\treturn 0;\n> -+}\n> -+\n> ++\treturn transaction;\n> + }\n> + \n> static int packed_transaction_prepare(struct ref_store *ref_store,\n> - \t\t\t\t struct ref_transaction *transaction,\n> - \t\t\t\t struct strbuf *err)\n> @@ refs/packed-backend.c: static int packed_transaction_prepare(struct ref_store *ref_store,\n> \t\t\tref_store,\n> \t\t\tREF_STORE_READ | REF_STORE_WRITE | REF_STORE_ODB,\n> @@ refs/packed-backend.c: static int packed_transaction_prepare(struct ref_store *r\n> \tif (write_with_updates(refs, &data->updates, err))\n> \t\tgoto failure;\n> \n> -@@ refs/packed-backend.c: struct ref_storage_be refs_be_packed = {\n> - \t.name = \"packed\",\n> - \t.init = packed_ref_store_create,\n> - \t.init_db = packed_init_db,\n> -+\t.transaction_begin = packed_transaction_begin,\n> - \t.transaction_prepare = packed_transaction_prepare,\n> - \t.transaction_finish = packed_transaction_finish,\n> - \t.transaction_abort = packed_transaction_abort,\n> \n> ## refs/packed-backend.h ##\n> @@ refs/packed-backend.h: struct ref_store *packed_ref_store_create(struct repository *repo,\n> @@ refs/packed-backend.h: struct ref_store *packed_ref_store_create(struct reposito\n> /*\n> * Return true if `transaction` really needs to be carried out against\n> * the specified packed_ref_store, or false if it can be skipped\n> -\n> - ## refs/refs-internal.h ##\n> -@@ refs/refs-internal.h: typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n> - \n> - typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n> - \n> -+typedef int ref_transaction_begin_fn(struct ref_store *refs,\n> -+\t\t\t\t struct ref_transaction *transaction,\n> -+\t\t\t\t struct strbuf *err);\n> -+\n> - typedef int ref_transaction_prepare_fn(struct ref_store *refs,\n> - \t\t\t\t struct ref_transaction *transaction,\n> - \t\t\t\t struct strbuf *err);\n> -@@ refs/refs-internal.h: struct ref_storage_be {\n> - \tref_store_init_fn *init;\n> - \tref_init_db_fn *init_db;\n> - \n> -+\tref_transaction_begin_fn *transaction_begin;\n> - \tref_transaction_prepare_fn *transaction_prepare;\n> - \tref_transaction_finish_fn *transaction_finish;\n> - \tref_transaction_abort_fn *transaction_abort;\n> 2: a5cb3adc662 ! 4: 0b8919b05c4 refs: move is_packed_transaction_needed out of packed-backend.c\n> @@ refs/files-backend.c: out:\n> \treturn ret;\n> }\n> \n> -+\n> +/*\n> + * Return true if `transaction` really needs to be carried out against\n> + * the specified packed_ref_store, or false if it can be skipped\n> @@ refs/files-backend.c: out:\n> struct files_transaction_backend_data {\n> \tstruct ref_transaction *packed_transaction;\n> \tint packed_transaction_needed;\n> -@@ refs/files-backend.c: struct ref_storage_be refs_be_files = {\n> - \t.delete_reflog = files_delete_reflog,\n> - \t.reflog_expire = files_reflog_expire\n> - };\n> -+\n> \n> ## refs/packed-backend.c ##\n> @@ refs/packed-backend.c: error:\n> 3: 559c04cee99 ! 5: d5669da57e3 refs: alternate reftable ref backend implementation\n> @@ Commit message\n> refs: alternate reftable ref backend implementation\n> \n> This introduces the reftable backend as an alternative to the packed\n> - backend. This simplifies the code, because we now longer have to worry\n> - about:\n> + backend.\n> +\n> + This is an alternative to the approach which was explored in\n> + https://github.com/git/git/pull/1215/. This alternative is simpler,\n> + because we now longer have to worry about:\n> \n> - pseudorefs and the HEAD ref\n> - worktrees\n> - commands that blur together files and references (cherry-pick, rebase)\n> \n> This deviates from the spec that in\n> - Documentation/technical/reftable.txt. It might be possible to update\n> - the code such that all writes by default go to reftable directly. Then\n> - the result would be compatible with an implementation that writes only\n> - reftable (the reftable lock would still prevent races) relative to an\n> - implementation that disregards loose refs. Or, JGit could be adapted\n> - to follow this implementation.\n> + Documentation/technical/reftable.txt. It might be possible to update the\n> + code such that all writes by default go to reftable directly. Then the\n> + result would be compatible with an implementation that writes only\n> + reftable (the reftable lock would still prevent races), but something\n> + must be done about the HEAD ref (which JGit keeps in reftable too.)\n> + Alternatively, JGit could be adapted to follow this implementation:\n> + despite the code being there for 4 years now, I haven't heard of anyone\n> + using it in production (exactly because CGit does not support it).\n> \n> For this incremental path, the reftable format is arguably more\n> complex than necessary, as\n> \n> - packed-refs doesn't support symrefs\n> - - reflogs aren't moved into reftable/\n> + - reflogs aren't moved into reftable\n> \n> on the other hand, the code is already there, and it's well-structured\n> and well-tested.\n> \n> - This implementation is a prototype. To test, you need to do `export\n> - GIT_TEST_REFTABLE=1`. Doing so creates a handful of errors in the\n> - test-suite, most seemingly related to the new behavior of pack-refs\n> - (which creates a reftable/ dir and not a packed-refs file.), but it\n> - seems overseeable.\n> + refs/reftable-backend.c was created by cannibalizing the first version\n> + of reftable support (github PR 1215 as mentioned above). It supports\n> + reflogs and symrefs, even though those features are never exercised.\n> +\n> + This implementation is a prototype, for the following reasons:\n> +\n> + - no considerations of backward compatibility and configuring an\n> + extension\n> + - no support for converting between packed-refs and reftable\n> + - no documentation\n> + - test failures when setting GIT_TEST_REFTABLE=1.\n> \n> Signed-off-by: Han-Wen Nienhuys \n> \n> @@ contrib/workdir/git-new-workdir: trap cleanup $siglist\n> \tcase $x in\n> \n> ## refs/files-backend.c ##\n> +@@\n> + #include \"../git-compat-util.h\"\n> ++#include \"../abspath.h\"\n> + #include \"../config.h\"\n> + #include \"../copy.h\"\n> + #include \"../environment.h\"\n> @@\n> #include \"refs-internal.h\"\n> #include \"ref-cache.h\"\n> @@ refs/files-backend.c\n> #include \"../iterator.h\"\n> #include \"../dir-iterator.h\"\n> @@ refs/files-backend.c: static struct ref_store *files_ref_store_create(struct repository *repo,\n> + \tstruct files_ref_store *refs = xcalloc(1, sizeof(*refs));\n> + \tstruct ref_store *ref_store = (struct ref_store *)refs;\n> + \tstruct strbuf sb = STRBUF_INIT;\n> ++\tint has_reftable, has_packed;\n> + \n> + \tbase_ref_store_init(ref_store, repo, gitdir, &refs_be_files);\n> \trefs->store_flags = flags;\n> \tget_common_dir_noenv(&sb, gitdir);\n> \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n> -+\n> -+\t/* TODO: should look at the repo to decide whether to use packed-refs or\n> -+\t * reftable. */\n> - \trefs->packed_ref_store =\n> +-\trefs->packed_ref_store =\n> -\t\tpacked_ref_store_create(repo, refs->gitcommondir, flags);\n> -+\t\tgit_env_bool(\"GIT_TEST_REFTABLE\", 0) \n> ++\n> ++\tstrbuf_addf(&sb, \"%s/reftable\", refs->gitcommondir);\n> ++\thas_reftable = is_directory(sb.buf);\n> ++\tstrbuf_reset(&sb);\n> ++\tstrbuf_addf(&sb, \"%s/packed-refs\", refs->gitcommondir);\n> ++\thas_packed = file_exists(sb.buf);\n> ++\n> ++\tif (!has_packed && !has_reftable)\n> ++\t\thas_reftable = git_env_bool(\"GIT_TEST_REFTABLE\", 0);\n> ++\n> ++\trefs->packed_ref_store = has_reftable\n> +\t\t? git_reftable_ref_store_create(repo, refs->gitcommondir, flags)\n> +\t\t: packed_ref_store_create(repo, refs->gitcommondir, flags);\n> \n> \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\n> \tchdir_notify_reparent(\"files-backend $GIT_COMMONDIR\",\n> \n> - ## refs/packed-backend.c ##\n> -@@ refs/packed-backend.c: static int write_packed_entry(FILE *fh, const char *refname,\n> - \treturn 0;\n> - }\n> - \n> --int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> -+static int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> - {\n> - \tstruct packed_ref_store *refs =\n> - \t\tpacked_downcast(ref_store, REF_STORE_WRITE | REF_STORE_MAIN,\n> -@@ refs/packed-backend.c: int packed_refs_lock(struct ref_store *ref_store, int flags, struct strbuf *err)\n> - \treturn 0;\n> - }\n> - \n> --void packed_refs_unlock(struct ref_store *ref_store)\n> -+static void packed_refs_unlock(struct ref_store *ref_store)\n> - {\n> - \tstruct packed_ref_store *refs = packed_downcast(\n> - \t\t\tref_store,\n> -@@ refs/packed-backend.c: void packed_refs_unlock(struct ref_store *ref_store)\n> - \trollback_lock_file(&refs->lock);\n> - }\n> - \n> --int packed_refs_is_locked(struct ref_store *ref_store)\n> --{\n> --\tstruct packed_ref_store *refs = packed_downcast(\n> --\t\t\tref_store,\n> --\t\t\tREF_STORE_READ | REF_STORE_WRITE,\n> --\t\t\t\"packed_refs_is_locked\");\n> --\n> --\treturn is_lock_file_locked(&refs->lock);\n> --}\n> --\n> - /*\n> - * The packed-refs header line that we write out. Perhaps other traits\n> - * will be added later.\n> -\n> ## refs/refs-internal.h ##\n> @@ refs/refs-internal.h: struct ref_storage_be {\n> };\n> @@ refs/reftable-backend.c (new)\n> +#include \"../worktree.h\"\n> +#include \"refs-internal.h\"\n> +#include \"reftable-backend.h\"\n> ++#include \"../repository.h\"\n> +\n> +extern struct ref_storage_be refs_be_reftable;\n> +\n> @@ refs/reftable-backend.c (new)\n> +\treturn &ri->base;\n> +}\n> +\n> -+static int fixup_symrefs(struct ref_store *ref_store,\n> -+\t\t\t struct ref_transaction *transaction)\n> -+{\n> -+\tstruct strbuf referent = STRBUF_INIT;\n> -+\tint i = 0;\n> -+\tint err = 0;\n> -+\n> -+\tfor (i = 0; i < transaction->nr; i++) {\n> -+\t\tstruct ref_update *update = transaction->updates[i];\n> -+\t\tstruct object_id old_oid;\n> -+\t\tint failure_errno;\n> -+\n> -+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n> -+\t\t\t\t\t\t&old_oid, &referent,\n> -+\t\t\t\t\t\t/* mutate input, like\n> -+\t\t\t\t\t\t files-backend.c */\n> -+\t\t\t\t\t\t&update->type, &failure_errno);\n> -+\t\tif (err < 0 && failure_errno == ENOENT &&\n> -+\t\t is_null_oid(&update->old_oid)) {\n> -+\t\t\terr = 0;\n> -+\t\t}\n> -+\t\tif (err < 0)\n> -+\t\t\tgoto done;\n> -+\n> -+\t\tif (!(update->type & REF_ISSYMREF))\n> -+\t\t\tcontinue;\n> -+\n> -+\t\tif (update->flags & REF_NO_DEREF) {\n> -+\t\t\t/* what should happen here? See files-backend.c\n> -+\t\t\t * lock_ref_for_update. */\n> -+\t\t} else {\n> -+\t\t\t/*\n> -+\t\t\t If we are updating a symref (eg. HEAD), we should also\n> -+\t\t\t update the branch that the symref points to.\n> -+\n> -+\t\t\t This is generic functionality, and would be better\n> -+\t\t\t done in refs.c, but the current implementation is\n> -+\t\t\t intertwined with the locking in files-backend.c.\n> -+\t\t\t*/\n> -+\t\t\tint new_flags = update->flags;\n> -+\t\t\tstruct ref_update *new_update = NULL;\n> -+\n> -+\t\t\t/* if this is an update for HEAD, should also record a\n> -+\t\t\t log entry for HEAD? See files-backend.c,\n> -+\t\t\t split_head_update()\n> -+\t\t\t*/\n> -+\t\t\tnew_update = ref_transaction_add_update(\n> -+\t\t\t\ttransaction, referent.buf, new_flags,\n> -+\t\t\t\t&update->new_oid, &update->old_oid,\n> -+\t\t\t\tupdate->msg);\n> -+\t\t\tnew_update->parent_update = update;\n> -+\n> -+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n> -+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n> -+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n> -+\t\t}\n> -+\t}\n> -+\n> -+done:\n> -+\tassert(err != REFTABLE_API_ERROR);\n> -+\tstrbuf_release(&referent);\n> -+\treturn err;\n> -+}\n> -+\n> -+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n> -+\t\t\t\t\t struct ref_transaction *transaction,\n> -+\t\t\t\t\t struct strbuf *errbuf)\n> ++static struct ref_transaction *git_reftable_transaction_begin(struct ref_store *ref_store,\n> ++\t\t\t\t\t\t\t struct strbuf *errbuf)\n> +{\n> +\tstruct git_reftable_ref_store *refs =\n> +\t\t(struct git_reftable_ref_store *)ref_store;\n> +\tstruct reftable_addition *add = NULL;\n> -+\tstruct reftable_stack *stack = stack_for(\n> -+\t\trefs,\n> -+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n> -+\tint i;\n> ++\tstruct ref_transaction *transaction = NULL;\n> ++\tstruct reftable_stack *stack = refs->main_stack;\n> +\n> +\tint err = refs->err;\n> +\tif (err < 0) {\n> @@ refs/reftable-backend.c (new)\n> +\n> +\terr = reftable_stack_new_addition(&add, stack);\n> +\tif (err) {\n> ++\t\tstrbuf_addf(errbuf, \"reftable: transaction begin: %s\",\n> ++\t\t\t reftable_error_str(err));\n> +\t\tgoto done;\n> +\t}\n> +\n> -+\tfor (i = 0; i < transaction->nr; i++) {\n> -+\t\tstruct ref_update *u = transaction->updates[i];\n> -+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n> -+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n> -+\t\t !(u->flags & REF_LOG_ONLY)) {\n> -+\t\t\tstruct object *o =\n> -+\t\t\t\tparse_object(refs->base.repo, &u->new_oid);\n> -+\t\t\tif (!o) {\n> -+\t\t\t\tstrbuf_addf(\n> -+\t\t\t\t\terrbuf,\n> -+\t\t\t\t\t\"trying to write ref '%s' with nonexistent object %s\",\n> -+\t\t\t\t\tu->refname, oid_to_hex(&u->new_oid));\n> -+\t\t\t\terr = -1;\n> -+\t\t\t\tgoto done;\n> -+\t\t\t}\n> -+\t\t}\n> -+\t}\n> -+\n> -+\terr = fixup_symrefs(ref_store, transaction);\n> -+\tif (err) {\n> -+\t\tgoto done;\n> -+\t}\n> -+\n> ++\tCALLOC_ARRAY(transaction, 1);\n> +\ttransaction->backend_data = add;\n> -+\ttransaction->state = REF_TRANSACTION_PREPARED;\n> -+\n> +done:\n> -+\tassert(err != REFTABLE_API_ERROR);\n> -+\tif (err < 0) {\n> -+\t\tif (add) {\n> -+\t\t\treftable_addition_destroy(add);\n> -+\t\t\tadd = NULL;\n> -+\t\t}\n> -+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n> -+\t\tif (!errbuf->len)\n> -+\t\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n> -+\t\t\t\t reftable_error_str(err));\n> -+\t}\n> ++\treturn transaction;\n> ++}\n> +\n> -+\treturn err;\n> ++static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n> ++\t\t\t\t\t struct ref_transaction *transaction,\n> ++\t\t\t\t\t struct strbuf *errbuf)\n> ++{\n> ++\ttransaction->state = REF_TRANSACTION_PREPARED;\n> ++\treturn 0;\n> +}\n> +\n> +static int git_reftable_transaction_abort(struct ref_store *ref_store,\n> @@ refs/reftable-backend.c (new)\n> +\t\t\t\t\t struct ref_transaction *transaction,\n> +\t\t\t\t\t struct strbuf *errmsg)\n> +{\n> ++\tstruct git_reftable_ref_store *refs =\n> ++\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n> +\tstruct reftable_addition *add =\n> +\t\t(struct reftable_addition *)transaction->backend_data;\n> +\tint err = 0;\n> @@ refs/reftable-backend.c (new)\n> +\n> +\terr = reftable_addition_commit(add);\n> +\n> ++\tif (!err)\n> ++\t\terr = reftable_stack_auto_compact(refs->main_stack);\n> +done:\n> +\tassert(err != REFTABLE_API_ERROR);\n> +\treftable_addition_destroy(add);\n> @@ refs/reftable-backend.c (new)\n> +\t.name = \"reftable\",\n> +\t.init = git_reftable_ref_store_create,\n> +\t.init_db = git_reftable_init_db,\n> ++\t.transaction_begin = git_reftable_transaction_begin,\n> +\t.transaction_prepare = git_reftable_transaction_prepare,\n> +\t.transaction_finish = git_reftable_transaction_finish,\n> +\t.transaction_abort = git_reftable_transaction_abort,\n> -: ----------- > 6: 2cf743031ab refs: always try to do packed transactions for reftable\n> \n> -- \n> gitgitgadget\n"}]}