{"thread":{"id":"56133","subject":"[PATCH 01/26] hash.h: provide constants for the hash IDs","startedAt":"2021-07-20T17:05:46Z","lastAt":"2021-08-30T14:11:12Z","messageCount":132,"participants":["Han-Wen Nienhuys via GitGitGadget","SZEDER Gábor via GitGitGadget","Junio C Hamano","Carlo Marcelo Arenas Belón","Han-Wen Nienhuys","Philip Oakley","Ævar Arnfjörð Bjarmason","Carlo Arenas"],"isPatch":true,"patchVersion":1,"patchTotal":26},"messages":[{"id":"430640","messageId":"dc24dccf041d78d9d50dcd25aa7f2cdfe960f916.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 01/26] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:21Z","receivedAt":"2021-07-20T17:05:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will simplify referencing them from code that is not deeply integrated with\nGit, in particular, the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n hash.h | 6 ++++++\n object-file.c | 7 ++-----\n 2 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9c6df4d9527..45b207f37cc 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -95,12 +95,18 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n /* Number of algorithms supported (including unknown). */\n #define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n+/* \"sha1\", big-endian */\n+#define GIT_SHA1_FORMAT_ID 0x73686131\n+\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* \"s256\", big-endian */\n+#define GIT_SHA256_FORMAT_ID 0x73323536\n+\n /* The length in bytes and in hex digits of an object name (SHA-256 value). */\n #define GIT_SHA256_RAWSZ 32\n #define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\ndiff --git a/object-file.c b/object-file.c\nindex ecca5a8da00..5f2b271e8bd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -164,7 +164,6 @@ static void git_hash_unknown_final_oid(struct object_id *oid, git_hash_ctx *ctx)\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n-\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\tNULL,\n@@ -183,8 +182,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha1\",\n-\t\t/* \"sha1\", big-endian */\n-\t\t0x73686131,\n+\t\tGIT_SHA1_FORMAT_ID,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n \t\tGIT_SHA1_BLKSZ,\n@@ -199,8 +197,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha256\",\n-\t\t/* \"s256\", big-endian */\n-\t\t0x73323536,\n+\t\tGIT_SHA256_FORMAT_ID,\n \t\tGIT_SHA256_RAWSZ,\n \t\tGIT_SHA256_HEXSZ,\n \t\tGIT_SHA256_BLKSZ,\n-- \ngitgitgadget\n\n"},{"id":"430641","messageId":"f1167669848563cbdd09dc90898edeff5ff59b2c.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 02/26] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:22Z","receivedAt":"2021-07-20T17:05:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 2167796ff2a..c2f03f6018e 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -425,6 +425,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"430642","messageId":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":null,"subject":"[PATCH 00/26] Support reftable ref backend for Git","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:20Z","receivedAt":"2021-07-20T17:05:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This continues the work in https://github.com/git/git/pull/847, which the\ngitgitgadget erroneously closed.\n\nChanges relative to last series (version 19 Apr 2021, tip: b8729fe)\n\n * drop EINVAL for broken refs.\n * fix 2 UBSAN warnings. Thanks to Andrzej Hunt for reporting.\n * make tests pass against master (use oidread)\n * handle worktree/xx/refname refs.\n * handle main-worktree/ref\n * fix double free in branch renaming\n * handle relative paths (used in git upload-pack invocation) for ref\n backend creation.\n * test and fix test-tool dump-reftable command\n\nFor the 'seen' branch, the following should be applied:\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d7137d1213..9323931eeb 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type);\n+ unsigned int *type, int *failure_errno);\n \n static void clear_reftable_log_record(struct reftable_log_record *log)\n {\n@@ -425,13 +424,14 @@ static int fixup_symrefs(struct ref_store *ref_store,\n for (i = 0; i < transaction->nr; i++) {\n struct ref_update *update = transaction->updates[i];\n struct object_id old_oid;\n+ int failure_errno;\n \n err = git_reftable_read_raw_ref(ref_store, update->refname,\n &old_oid, &referent,\n /* mutate input, like\n files-backend.c */\n- &update->type);\n- if (err < 0 && errno == ENOENT &&\n+ &update->type, &failure_errno);\n+ if (err < 0 && failure_errno == ENOENT &&\n is_null_oid(&update->old_oid)) {\n err = 0;\n }\n@@ -1602,7 +1603,7 @@ static int reftable_error_to_errno(int err)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type)\n+ unsigned int *type, int *failure_errno)\n {\n struct git_reftable_ref_store *refs =\n (struct git_reftable_ref_store *)ref_store;\n@@ -1626,13 +1627,11 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \n err = reftable_stack_read_ref(stack, refname, &ref);\n if (err > 0) {\n- errno = ENOENT;\n+ *failure_errno = ENOENT;\n err = -1;\n goto done;\n }\n if (err < 0) {\n- errno = reftable_error_to_errno(err);\n- err = -1;\n goto done;\n }\n\n\nHan-Wen Nienhuys (25):\n hash.h: provide constants for the hash IDs\n init-db: set the_repository->hash_algo early on\n reftable: RFC: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n Provide zlib's uncompress2 from compat/zlib-compat.c\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n refs: RFC: Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n t1301: document what needs to be done for reftable\n t1401,t2011: parameterize HEAD.lock for REFFILES\n t1404: annotate test cases with REFFILES\n t7004: avoid direct filesystem access\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 54 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 60 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +\n config.mak.uname | 2 +-\n configure.ac | 13 +\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n hash.h | 6 +\n object-file.c | 7 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1683 +++++++++++++++++\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 +\n reftable/block.c | 446 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 105 +\n reftable/error.c | 41 +\n reftable/generic.c | 169 ++\n reftable/generic.h | 32 +\n reftable/iter.c | 194 ++\n reftable/iter.h | 69 +\n reftable/merged.c | 362 ++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 292 +++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 801 ++++++++\n reftable/reader.h | 66 +\n reftable/readwrite_test.c | 650 +++++++\n reftable/record.c | 1200 ++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 407 ++++\n reftable/refname.c | 209 ++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 47 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 101 +\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 128 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 148 ++\n reftable/reftable.c | 115 ++\n reftable/stack.c | 1396 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 947 ++++++++++\n reftable/system.h | 24 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 690 +++++++\n reftable/writer.h | 50 +\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 6 +\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 297 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1301-shared-repo.sh | 8 +-\n t/t1401-symbolic-ref.sh | 11 +-\n t/t1404-update-ref-errors.sh | 56 +-\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t2011-checkout-invalid-head.sh | 11 +-\n t/t3210-pack-refs.sh | 6 +\n t/t7004-tag.sh | 7 +-\n t/test-lib.sh | 7 +-\n 94 files changed, 13129 insertions(+), 63 deletions(-)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/readwrite_test.c\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: daab8a564f8bbac55f70f8bf86c070e001a9b006\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1054%2Fhanwen%2Flibreftable-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1054/hanwen/libreftable-v1\nPull-Request: https://github.com/git/git/pull/1054\n-- \ngitgitgadget\n"},{"id":"430643","messageId":"76c2a1005da68babc292eedf05daa29d7f236fcf.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 03/26] reftable: RFC: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:23Z","receivedAt":"2021-07-20T17:06:16Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe objective of this code is to be usable as a C library, so it can be reused\nin libgit2.\n\nThis is currently using a BSD license as it is the liberal license I could find,\nbut this could be changed to whatever fits the stated goal above.\n\nThis code is currently imported from github.com/hanwen/reftable. Once this code\nlands in git.git, the C code will be removed from github.com/hanwen/reftable,\nand the git.git code will be the source of truth.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 00000000000..402e0f9356b\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"430644","messageId":"5ef4b7040a360f60f33dbe6f30afe1db4e54b384.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 06/26] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:26Z","receivedAt":"2021-07-20T17:06:18Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex ed969b0793f..ad10ada9283 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2443,6 +2443,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 00000000000..0044eecd9aa\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(!bs->ops);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(!bs->ops);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 00000000000..072e2727ad2\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 00000000000..5aa3990a573\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"430645","messageId":"616d6ed89eebe1a9ba55c51656d4426f07b7b153.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 04/26] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:24Z","receivedAt":"2021-07-20T17:06:18Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 00000000000..f6f16def921\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 00000000000..6f89bedf1a5\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"430646","messageId":"9d2cdfe3ddd645d5d82f30db184538ad208114f9.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 05/26] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:25Z","receivedAt":"2021-07-20T17:06:18Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 24 ++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 545 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex c7c46c017d3..ed969b0793f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -741,6 +741,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -819,6 +820,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1191,7 +1194,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2438,7 +2441,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2454,6 +2465,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2604,6 +2617,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2888,7 +2907,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3218,7 +3237,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 171b4124afe..c2bf5bdffc6 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -640,6 +640,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -662,7 +668,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -902,11 +908,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 00000000000..f761e48028c\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (!a) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 00000000000..096b36862b9\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 00000000000..1fcd2297256\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(!out[2]);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!out[1]);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 00000000000..bd0a02d3f68\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 00000000000..5f2185f1f34\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 00000000000..5e7698ae654\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 00000000000..bf963ee458e\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,24 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+// This header glues the reftable library to the rest of Git\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hash.h\" /* hash ID, sizes.*/\n+#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+\n+#include \n+\n+struct strbuf;\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 00000000000..84ac972cad0\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_FORMAT_ID));\n+}\n+\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add(b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 00000000000..774cb275bf6\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 00000000000..3b58e423e7b\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex b21e8f15190..01201629fca 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -51,13 +51,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex f845ced4b3a..cb90b7f4f7b 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -47,6 +47,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 00000000000..0ed14971a58\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"430647","messageId":"27abd15965875115f641ed9e0cb1a8732369eddb.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 08/26] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:28Z","receivedAt":"2021-07-20T17:06:39Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be needed for reading reflog blocks in reftable.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 7 +++\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n configure.ac | 13 ++++++\n 5 files changed, 114 insertions(+)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n\ndiff --git a/Makefile b/Makefile\nindex 15321edbd2c..640a332b481 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -256,6 +256,8 @@ all::\n #\n # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n #\n+# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n+#\n # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n #\n@@ -1734,6 +1736,11 @@ ifdef NO_DEFLATE_BOUND\n \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n endif\n \n+ifdef NO_UNCOMPRESS2\n+\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n+\tLIB_OBJS += compat/zlib-uncompress2.o\n+endif\n+\n ifdef NO_POSIX_GOODIES\n \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n endif\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..5711c63979d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -224,6 +224,7 @@ linux-gcc-default)\n \t;;\n Linux32)\n \tCC=gcc\n+\tMAKEFLAGS=\"$MAKEFLAGS NO_UNCOMPRESS2=1\"\n \t;;\n linux-musl)\n \tCC=gcc\ndiff --git a/compat/.gitattributes b/compat/.gitattributes\nnew file mode 100644\nindex 00000000000..40dbfb170da\n--- /dev/null\n+++ b/compat/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\nnew file mode 100644\nindex 00000000000..6893bb469ce\n--- /dev/null\n+++ b/compat/zlib-uncompress2.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress2 (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/configure.ac b/configure.ac\nindex 031e8d3fee8..c3a913103d0 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n LIBS=\"$old_LIBS\"\n \n+AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n+AC_LANG_PROGRAM([#include ],\n+ [uncompress2(NULL,NULL,NULL,NULL);])])\n+AC_MSG_CHECKING([for uncompress2 in -lz])\n+old_LIBS=\"$LIBS\"\n+LIBS=\"$LIBS -lz\"\n+AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n+\t[AC_MSG_RESULT([yes])],\n+\t[AC_MSG_RESULT([no])\n+\tNO_UNCOMPRESS2=yes])\n+LIBS=\"$old_LIBS\"\n+\n GIT_UNSTASH_FLAGS($ZLIB_PATH)\n \n GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n+GIT_CONF_SUBST([NO_UNCOMPRESS2])\n \n #\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n-- \ngitgitgadget\n\n"},{"id":"430648","messageId":"13a5cbef0df1a6efb4309b1e9c2f76dcf66ed54b.1626800686.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 07/26] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:27Z","receivedAt":"2021-07-20T17:06:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1200 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 407 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1884 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex ad10ada9283..15321edbd2c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2445,7 +2445,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 00000000000..5eee72c4c11\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 00000000000..34ed480b257\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1200 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = rec;\n+\tconst struct reftable_ref_record *src = src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release(rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tobj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));\n+\tCOPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->update.name,\n+\t\t log->update.email, log->update.time,\n+\t\t log->update.tz_offset);\n+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->update.email) {\n+\t\t\tdst->update.email = xstrdup(dst->update.email);\n+\t\t}\n+\t\tif (dst->update.name) {\n+\t\t\tdst->update.name = xstrdup(dst->update.name);\n+\t\t}\n+\t\tif (dst->update.message) {\n+\t\t\tdst->update.message = xstrdup(dst->update.message);\n+\t\t}\n+\n+\t\tif (dst->update.new_hash) {\n+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tif (dst->update.old_hash) {\n+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->update.new_hash);\n+\t\treftable_free(r->update.old_hash);\n+\t\treftable_free(r->update.name);\n+\t\treftable_free(r->update.email);\n+\t\treftable_free(r->update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_log_record *r = rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->update.old_hash;\n+\tnewh = r->update.new_hash;\n+\tif (!oldh) {\n+\t\toldh = zero;\n+\t}\n+\tif (!newh) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->update.old_hash);\n+\t\t\tFREE_AND_NULL(r->update.new_hash);\n+\t\t\tFREE_AND_NULL(r->update.message);\n+\t\t\tFREE_AND_NULL(r->update.email);\n+\t\t\tFREE_AND_NULL(r->update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n+\n+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n+\tmemcpy(r->update.name, dest.buf, dest.len);\n+\tr->update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n+\tmemcpy(r->update.email, dest.buf, dest.len);\n+\tr->update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n+\tmemcpy(r->update.message, dest.buf, dest.len);\n+\tr->update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (!a)\n+\t\ta = empty;\n+\n+\tif (!b)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (!a)\n+\t\ta = zero;\n+\n+\tif (!b)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->update.name, b->update.name) &&\n+\t\t a->update.time == b->update.time &&\n+\t\t a->update.tz_offset == b->update.tz_offset &&\n+\t\t null_streq(a->update.email, b->update.email) &&\n+\t\t null_streq(a->update.message, b->update.message) &&\n+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n+\t\t\t\t hash_size) &&\n+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n+\t\t\t\t hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_index_record *rec = r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = rec;\n+\tconst struct reftable_index_record *src = src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a && b)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tconst struct reftable_log_record *la = a;\n+\tconst struct reftable_log_record *lb = b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 00000000000..498e8c50bf4\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 00000000000..bf5d072b20d\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,407 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t.time = 1577123507,\n+\t\t\t\t.tz_offset = 100,\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].update.new_hash, 1);\n+\tset_test_hash(in[0].update.old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 00000000000..7985b94ae2c\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t};\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b..09d4b83ef9b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"430649","messageId":"92970da9cb9105913753211c2b3bb98c1c577bc6.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 10/26] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:30Z","receivedAt":"2021-07-20T17:07:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 91be4b9c27a..12bd12328b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,11 +2454,13 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 00000000000..82db7995dd6\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 00000000000..fbdd002e23a\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 00000000000..09a970e17b9\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = arg;\n+\tif (c->last) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7..050551fa698 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"430650","messageId":"fa9d3b95fd441bd9d190362e1d7c26d3ad7cbb0b.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 13/26] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:33Z","receivedAt":"2021-07-20T17:07:35Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 194 +++++++++\n reftable/iter.h | 69 ++++\n reftable/reader.c | 801 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 +++\n reftable/reftable-reader.h | 101 +++++\n 6 files changed, 1233 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex 9e0aefd96f7..ef4bf0f6a49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2452,7 +2452,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/generic.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 00000000000..93d04f735b8\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,194 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn !it->ops;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\treftable_free(it->offsets);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 00000000000..09eb0cbfa59\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,69 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+\n+/* Takes ownership of `offsets` */\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 00000000000..49f4ec070e6\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,801 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (!source->ops) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\tuint64_t file_size = block_source_size(source);\n+\n+\t/* Need +1 to read type of first block. */\n+\tuint32_t read_size = header_size(2) + 1; /* read v2 because it's larger. */\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\tif (read_size > file_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = block_source_read_block(source, &header, 0, read_size);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = file_size - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (!ti->bi.br) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next(ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek(tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id(tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+\n+int reftable_reader_print_file(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_reader *r = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\treftable_table_from_reader(&tab, r);\n+\terr = reftable_table_print(&tab);\n+done:\n+\treftable_reader_free(r);\n+\treturn err;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 00000000000..39583e5dbcd\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 00000000000..4a4bc2fdf85\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,101 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* print table onto stdout for debugging. */\n+int reftable_reader_print_file(const char *tablename);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"430651","messageId":"4d7745e65529c3e77cc6d016ca75084e265c96fb.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 15/26] reftable: add a heap-based priority queue for reftable records","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:35Z","receivedAt":"2021-07-20T17:07:35Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 235d30b55d6..2123aa782c9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,6 +2454,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n@@ -2464,6 +2465,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 00000000000..8918d158e2d\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 00000000000..385d2fb139a\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 00000000000..ad21673e854\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 3d541fa5c0c..0019cbcfa49 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ https://developers.google.com/open-source/licenses/bsd\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int readwrite_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 898aba836fd..0b5a1701df1 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"430652","messageId":"25aa2bf9b71c80b513bab8183ffad322c1a9444e.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 09/26] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:29Z","receivedAt":"2021-07-20T17:07:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/block.c | 446 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 697 insertions(+)\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 640a332b481..91be4b9c27a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2450,10 +2450,12 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 00000000000..92f8e5abfad\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,446 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \n+\n+#ifdef NO_UNCOMPRESS2\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n+\t\tuLong *sourceLen);\n+#endif\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK !=\n+\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n+\t\t\t\tblock->data + block_header_skip, &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp && source.ops)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 00000000000..e207706a644\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 00000000000..c3d35eedb98\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b..c9deeaf08c7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"430653","messageId":"99708f408b0f51c12741ce5a486476aeb895bc2a.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 12/26] reftable: generic interface to tables","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:32Z","receivedAt":"2021-07-20T17:07:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/generic.c | 169 +++++++++++++++++++++++++++++++++++\n reftable/generic.h | 32 +++++++\n reftable/reftable-generic.h | 47 ++++++++++\n reftable/reftable-iterator.h | 39 ++++++++\n reftable/reftable.c | 115 ++++++++++++++++++++++++\n 6 files changed, 405 insertions(+)\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex af553fc227a..9e0aefd96f7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,6 +2454,9 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n+REFTABLE_OBJS += reftable/generic.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/generic.c b/reftable/generic.c\nnew file mode 100644\nindex 00000000000..7a8a738d860\n--- /dev/null\n+++ b/reftable/generic.c\n@@ -0,0 +1,169 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = ~((uint64_t)0),\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_print(struct reftable_table *tab) {\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tuint32_t hash_id = reftable_table_hash_id(tab);\n+\tint err = reftable_table_seek_ref(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_table_seek_log(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn 0;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 00000000000..98886a06402\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 00000000000..d239751a778\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,47 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* dump table contents onto stdout for debugging */\n+int reftable_table_print(struct reftable_table *tab);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 00000000000..d3eee7af357\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 00000000000..0e4607a7cd6\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \ngitgitgadget\n\n"},{"id":"430654","messageId":"3eeefa5665f2b9e6c89e78584d3912bdf0362ae4.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 14/26] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:34Z","receivedAt":"2021-07-20T17:07:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/readwrite_test.c | 650 ++++++++++++++++++++++++++++++++++++++\n reftable/reftable-tests.h | 2 +-\n t/helper/test-reftable.c | 1 +\n 4 files changed, 653 insertions(+), 1 deletion(-)\n create mode 100644 reftable/readwrite_test.c\n\ndiff --git a/Makefile b/Makefile\nindex ef4bf0f6a49..235d30b55d6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2465,6 +2465,7 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nnew file mode 100644\nindex 00000000000..42ca48f83c4\n--- /dev/null\n+++ b/reftable/readwrite_test.c\n@@ -0,0 +1,650 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.new_hash = hash;\n+\t\tlog.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == GIT_SHA256_FORMAT_ID) ? 2 :\n+\t\t\t\t\t\t\t\t\t\t 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n+\t\t\t\t\t .update_index = 0xa,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t\t } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\tfor (int i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.update.old_hash = hash1;\n+\tlog.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.old_hash = hash1;\n+\t\tlog.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n+\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_write_empty_table(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_write_key_order(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record refs[2] = {\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}, {\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}\n+\t};\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\terr = reftable_writer_add_ref(w, &refs[0]);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_add_ref(w, &refs[1]);\n+\tprintf(\"%d\\n\", err);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_writer_close(w);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_corrupt_table_empty(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+}\n+\n+static void test_corrupt_table(void)\n+{\n+\tuint8_t zeros[1024] = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\tstrbuf_add(&buf, zeros, sizeof(zeros));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\tstrbuf_release(&buf);\n+}\n+\n+int readwrite_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_corrupt_table);\n+\tRUN_TEST(test_corrupt_table_empty);\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_write_key_order);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_write_empty_table);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654..3d541fa5c0c 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n-int reftable_test_main(int argc, const char **argv);\n+int readwrite_test_main(int argc, const char **argv);\n int stack_test_main(int argc, const char **argv);\n int tree_test_main(int argc, const char **argv);\n int reftable_dump_main(int argc, char *const *argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa698..898aba836fd 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"430655","messageId":"3b817f37a01def9337ae60107a3e082cf8da5937.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 11/26] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:31Z","receivedAt":"2021-07-20T17:07:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 148 ++++++++\n reftable/writer.c | 690 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 12bd12328b5..af553fc227a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2455,6 +2455,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 00000000000..af36462ced5\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \"reftable-record.h\"\n+\n+#include \n+#include /* ssize_t */\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 00000000000..1baad069b64\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,690 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\t\t\t 1 :\n+\t\t\t 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy(dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = -1;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->update.message;\n+\tif (!w->opts.exact_log_message && log->update.message) {\n+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tif (arg->last) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 00000000000..09b88673d97\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tssize_t (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"430656","messageId":"05e46f7e1d8ec53a80740c312234065eb65e3fe0.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 19/26] reftable: add dump utility","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:39Z","receivedAt":"2021-07-20T17:07:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/dump.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 105 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 00000000000..668cfa89965\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,105 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n+#include \"reftable-generic.h\"\n+#include \"hash.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -6 sha256 hash format\\n\"\n+\t \" -h this help\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n+\t\t\tbreak;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n+\t\t\topt_dump_table = 1;\n+\t\telse if (!strcmp(\"-6\", argv[1]))\n+\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n+\t\t\topt_dump_stack = 1;\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n+\t\t\topt_compact = 1;\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t}\n+\n+\tif (argc != 2) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[1];\n+\n+\tif (opt_dump_table) {\n+\t\terr = reftable_reader_print_file(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = reftable_stack_print_directory(arg, opt_hash_id);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \ngitgitgadget\n\n"},{"id":"430657","messageId":"806c2e04392fa4af7f3d5abf2d1e4401898ab29f.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 17/26] reftable: implement refname validation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:37Z","receivedAt":"2021-07-20T17:08:09Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 342 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 9369013daed..06a5fb9103b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2470,6 +2470,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 00000000000..95734969324\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 00000000000..a24b40fcb42\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 00000000000..8645cd93bbd\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 8087f2da4e6..c8db6852c35 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"430658","messageId":"c906d79b341f1e6410598d2a54a3341705822b9c.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 16/26] reftable: add merged table view","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:36Z","receivedAt":"2021-07-20T17:08:09Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 764 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 2123aa782c9..9369013daed 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,6 +2454,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2465,6 +2466,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 00000000000..e5b53da6db3\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,362 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (!mt) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record(tab, it, rec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 00000000000..8c4d4d58d77\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 00000000000..1e2afe37b8b\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,292 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n+\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == GIT_SHA1_FORMAT_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 00000000000..1a6d16915ab\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 0b5a1701df1..8087f2da4e6 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"430659","messageId":"d1d4b2f9e9b59e208bb36bd6050f10884c8be6a1.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 20/26] refs: RFC: Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:40Z","receivedAt":"2021-07-20T17:09:13Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see Documentation/technical/reftable.txt.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by setting GIT_TEST_REFTABLE in the environment. When\nGIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set.\n\nThere is no option to git-init for now, as the test suite still shows failures\nwith GIT_TEST_REFTABLE=1.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nHelped-by: Patrick Steinhardt \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 39 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1683 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 6 +\n t/t0031-reftable.sh | 291 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 22 files changed, 2129 insertions(+), 28 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdca..82c5940f143 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffd..72576235833 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex c18042929c8..19566c661f1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -986,6 +986,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66fe66679c8..baa1ff4fc60 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c2f03f6018e..22b07d2b2fb 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -226,6 +228,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -235,6 +238,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -249,9 +270,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -268,7 +286,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -383,7 +401,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -422,6 +440,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -476,6 +495,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -549,6 +571,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -713,5 +736,5 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..d43d6898039 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n static int do_clear_stash(void)\n {\n \tstruct object_id obj;\n+\tint result;\n \tif (get_oid(ref_stash, &obj))\n \t\treturn 0;\n \n-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n+\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n+\n+\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n+\t * even when refs are deleted. */\n+\tdelete_reflog(ref_stash);\n+\treturn result;\n }\n \n static int clear_stash(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 976bf8ed063..9601ccd5909 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -328,9 +329,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd36..f4472afb3fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -647,9 +647,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1067,6 +1068,7 @@ struct repository_format {\n \tint hash_algo;\n \tint sparse_index;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 69413fb3dc0..a2b156e338e 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -711,7 +711,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba1..1a25789d285 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ EOM\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ EOM\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex 8b9f7c3a80a..00c4d24df6e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,15 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1880,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1902,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1962,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1976,7 @@ done:\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1990,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0..5a6d4ca9fa8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 3155708345f..e36f215067e 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -672,6 +672,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..d7137d12138\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1683 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * two separate databases (ie. two reftable/ directories), one for the\n+ * repository, and one for the worktree. For reading, we merge the view (see\n+ * git_reftable_iterator) of both, when necessary.\n+ *\n+ * Unfortunately, the worktrees can also be selected by specifying a magic\n+ * refname (eg. worktree/BLA/refname, even if BLA isn't the current worktree.)\n+ */\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tconst char *wtname = refname;\n+\tint wtname_len = 0;\n+\tconst char *wtref = refname;\n+\n+\tif (refname == NULL)\n+\t\treturn store->main_stack;\n+\n+\tif (!parse_worktree_ref(refname, &wtname, &wtname_len, &wtref) &&\n+\t wtname_len) {\n+\t\t/* this makes me cry. Woe you if you try to access\n+\t\t * worktree/BLA/REF and the current worktree\n+\t\t * from the same process.\n+\t\t */\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_write_options cfg = {\n+\t\t\t.block_size = 4096,\n+\t\t\t.hash_id = the_hash_algo->format_id,\n+\t\t};\n+\n+\t\tstrbuf_addstr(&wt_dir, store->base.gitdir);\n+\t\tstrbuf_addstr(&wt_dir, \"/worktrees/\");\n+\t\tstrbuf_add(&wt_dir, wtname, wtname_len);\n+\t\tstrbuf_addstr(&wt_dir, \"/reftable\");\n+\n+\t\tif (store->worktree_stack)\n+\t\t\treftable_stack_destroy(store->worktree_stack);\n+\t\tstore->err = reftable_new_stack(&store->worktree_stack,\n+\t\t\t\t\t\twt_dir.buf, cfg);\n+\t\tassert(store->err != REFTABLE_API_ERROR);\n+\n+\t\treturn store->worktree_stack;\n+\t}\n+\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *out = ref;\n+\tint name_len = 0;\n+\tif (skip_prefix(ref, \"main-worktree/\", &out))\n+\t\treturn out;\n+\n+\tif (!parse_worktree_ref(ref, NULL, &name_len, &out) && name_len) {\n+\t\treturn out;\n+\t}\n+\n+\treturn ref;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->update.old_hash = NULL;\n+\t\tlog->update.new_hash = NULL;\n+\t\tlog->update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path. Mutates its arg, trimming it to the\n+ * gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_realpath(&wt_buf, path, /*die_on_error=*/0);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\n+\t\trefs->err =\n+\t\t\treftable_new_stack(&refs->worktree_stack, sb.buf, cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs,\n+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_id = *null_oid();\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->update.new_hash = u->new_oid.hash;\n+\t\tlog->update.message = u->msg;\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\n+\t\tlog->update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.update.message = xstrdup(arg->logmsg);\n+\t\tlog.update.new_hash = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.update.old_hash != NULL ||\n+\t\t log.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.update.message = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tlog.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\told_ref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].update.message = (char *)arg->logmsg;\n+\t\ttodo[0].update.old_hash = val1;\n+\t\ttodo[0].update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].update.old_hash = NULL;\n+\t\ttodo[1].update.new_hash = val1;\n+\t\ttodo[1].update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* this copies the entire reflog history. Is this the right semantics?\n+\t */\n+\t/* XXX should clear out existing reflog entries for oldname? */\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\toidread(&ri->oid, ri->log.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.update.old_hash);\n+\t\toidread(&new_oid, log.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->update.old_hash);\n+\t\toidread(&new_oid, log->update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].update.old_hash);\n+\t\toidread(&noid, logs[i].update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].update.email,\n+\t\t\t\t (timestamp_t)logs[i].update.time,\n+\t\t\t\t logs[i].update.tz_offset,\n+\t\t\t\t logs[i].update.message, policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash != NULL) {\n+\t\t\t\tdest->update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* XXX - skip writing records that were not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t} else {\n+\t\t/* XXX - print something */\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex b2bf44c6faf..f1cc8df47c4 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -180,6 +180,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex 3740c93bc0f..1bd9b4d09c8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -82,6 +82,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex eb9367ca5cb..72d27477560 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -498,6 +498,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -648,6 +651,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1312,6 +1316,8 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\tthe_repository->repository_format_partial_clone =\n \t\t\t\trepo_fmt.partial_clone;\n \t\t\trepo_fmt.partial_clone = NULL;\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n \t\t}\n \t}\n \t/*\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 00000000000..7899a1c580e\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,291 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+write_script fake_editor <<\\EOF\n+echo \"$MSG\" >\"$1\"\n+echo \"$MSG\" >&2\n+EOF\n+GIT_EDITOR=./fake_editor\n+export GIT_EDITOR\n+\n+\n+test_expect_success 'using reftable' '\n+\tinitialize &&\n+\ttest -d .git/reftable &&\n+\ttest -f .git/reftable/tables.list\n+'\n+\n+test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n+\tinitialize &&\n+\ttest_commit 1st &&\n+\ttest_commit 2nd &&\n+\tMSG=b4 git notes add &&\n+\tMSG=b3 git notes edit &&\n+\techo b4 >expect &&\n+\tgit notes --ref commits@{1} show >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git reflog delete' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_commit file2 &&\n+\ttest_commit file3 &&\n+\ttest_commit file4 &&\n+\tgit reflog delete HEAD@{1} &&\n+\tgit reflog > output &&\n+\t! grep file3 output\n+'\n+\n+test_expect_success 'branch -D delete nonexistent branch' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_must_fail git branch -D ../../my-private-file\n+'\n+\n+test_expect_success 'branch copy' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\ttest_commit file2 &&\n+\tgit branch src &&\n+\tgit reflog src > expect &&\n+\tgit branch -c src dst &&\n+\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'update-ref on corrupted data' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tOLD_SHA1=$(git rev-parse HEAD) &&\n+\ttest_commit file2 &&\n+\tls -l .git/reftable &&\n+\tfor f in .git/reftable/*.ref\n+\tdo\n+\t\t>$f\n+\tdone &&\n+\ttest_must_fail git update-ref refs/heads/main $OLD_SHA1\n+'\n+\n+test_expect_success 'git stash' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttouch actual expected &&\n+\tgit -c status.showStash=true status >expected &&\n+\techo hoi >> file.t &&\n+\tgit stash push -m stashed &&\n+\tgit stash clear &&\n+\tgit -c status.showStash=true status >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'rename branch' '\n+\tinitialize &&\n+\tgit symbolic-ref HEAD refs/heads/before &&\n+\ttest_commit file &&\n+\tgit show-ref | sed s/before/after/g > expected &&\n+\tgit branch -M after &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(GIT_TEST_REFTABLE=1 git_init || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\t(GIT_TEST_REFTABLE=1 git_init start) &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506..cdc21bf2dcb 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; need files backend'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..6bdd430dfe3 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 577f32dc71f..e523c3dd624 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; requires files ref backend'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9e268605449..c9b06f931b4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1509,7 +1509,12 @@ parisc* | hppa*)\n \t;;\n esac\n \n-test_set_prereq REFFILES\n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq !REFFILES\n+else\n+ test_set_prereq REFFILES\n+fi\n \n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n-- \ngitgitgadget\n\n"},{"id":"430660","messageId":"d895736f0e1b96fa407252acaaaf962ed1cfb667.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 24/26] t1401,t2011: parameterize HEAD.lock for REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:44Z","receivedAt":"2021-07-20T17:09:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1401-symbolic-ref.sh | 11 +++++++++--\n t/t2011-checkout-invalid-head.sh | 11 +++++++++--\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh\nindex 132a1b885ac..1b51013aded 100755\n--- a/t/t1401-symbolic-ref.sh\n+++ b/t/t1401-symbolic-ref.sh\n@@ -102,9 +102,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' '\n \ttest_cmp expect actual\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'symbolic-ref reports failure in exit code' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git symbolic-ref HEAD refs/heads/whatever\n '\n \ndiff --git a/t/t2011-checkout-invalid-head.sh b/t/t2011-checkout-invalid-head.sh\nindex e52022e1522..a56f7af442c 100755\n--- a/t/t2011-checkout-invalid-head.sh\n+++ b/t/t2011-checkout-invalid-head.sh\n@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' '\n \tgit checkout main --\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'checkout notices failure to lock HEAD' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git checkout -b other\n '\n \n-- \ngitgitgadget\n\n"},{"id":"430661","messageId":"73eece0caacb7519edf338f040f08d381456df88.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 23/26] t1301: document what needs to be done for reftable","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:43Z","receivedAt":"2021-07-20T17:09:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1301-shared-repo.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh\nindex 84bf1970d8b..a5755b4a434 100755\n--- a/t/t1301-shared-repo.sh\n+++ b/t/t1301-shared-repo.sh\n@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' '\n \t)\n '\n \n+# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/\n for u in 002 022\n do\n-\ttest_expect_success POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n+\ttest_expect_success REFFILES,POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n \t\tmkdir sub && (\n \t\t\tcd sub &&\n \t\t\tumask $u &&\n@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' '\n+# For reftable, the check on .git/reftable/ is sufficient.\n+test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' '\n \tumask 077 &&\n \tgit config core.sharedRepository group &&\n \tgit reflog expire --all &&\n@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)'\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'template can set core.sharedrepository' '\n+test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' '\n \trm -rf child.git &&\n \tumask 0022 &&\n \tgit config core.sharedrepository 0666 &&\n-- \ngitgitgadget\n\n"},{"id":"430662","messageId":"4a5891fa8cc9514535d70b52a4fce2b21991bfef.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 26/26] t7004: avoid direct filesystem access","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:46Z","receivedAt":"2021-07-20T17:09:30Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t7004-tag.sh | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..8bd84b0e404 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -97,7 +97,8 @@ test_expect_success 'creating a tag with --create-reflog should create reflog' '\n \ttest_when_finished \"git tag -d tag_with_reflog\" &&\n \tgit tag --create-reflog tag_with_reflog &&\n \tgit reflog exists refs/tags/tag_with_reflog &&\n-\tsed -e \"s/^.*\t//\" .git/logs/refs/tags/tag_with_reflog >actual &&\n+\tgit reflog --format=\"format:tag: tagging %h (%s, %cd)%n\" \\\n+\t\t--date=format:%Y-%m-%d refs/tags/tag_with_reflog >actual &&\n \ttest_cmp expected actual\n '\n \n@@ -108,7 +109,9 @@ test_expect_success 'annotated tag with --create-reflog has correct message' '\n \ttest_when_finished \"git tag -d tag_with_reflog\" &&\n \tgit tag -m \"annotated tag\" --create-reflog tag_with_reflog &&\n \tgit reflog exists refs/tags/tag_with_reflog &&\n-\tsed -e \"s/^.*\t//\" .git/logs/refs/tags/tag_with_reflog >actual &&\n+\tgit reflog \\\n+\t\t--format=\"format:tag: tagging %h (%s, %cd)%n\" \\\n+\t\t--date=format:%Y-%m-%d >actual &&\n \ttest_cmp expected actual\n '\n \n-- \ngitgitgadget\n"},{"id":"430663","messageId":"f0a528b9ece4ec964f7698843613ae025f2b5f5d.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 25/26] t1404: annotate test cases with REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:45Z","receivedAt":"2021-07-20T17:09:30Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\n* Reftable for now lacks detailed error messages for directory/file conflicts.\n Skip message comparisons.\n\n* Mark tests that muck with .git directly as REFFILES.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1404-update-ref-errors.sh | 56 +++++++++++++++++++++++++++---------\n 1 file changed, 42 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t1404-update-ref-errors.sh b/t/t1404-update-ref-errors.sh\nindex b729c1f4803..811d5bb56d4 100755\n--- a/t/t1404-update-ref-errors.sh\n+++ b/t/t1404-update-ref-errors.sh\n@@ -27,7 +27,9 @@ test_update_rejected () {\n \tfi &&\n \tprintf \"create $prefix/%s $C\\n\" $create >input &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_i18ngrep -F \"$error\" output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_i18ngrep -F \"$error\" output.err\n+\tfi &&\n \tgit for-each-ref $prefix >actual &&\n \ttest_cmp unchanged actual\n }\n@@ -101,7 +103,9 @@ df_test() {\n \t\tprintf \"%s\\n\" \"delete $delname\" \"create $addname $D\"\n \tfi >commands &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_cmp expected-err output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected-err output.err\n+\tfi &&\n \tprintf \"%s\\n\" \"$C $delref\" >expected-refs &&\n \tgit for-each-ref --format=\"%(objectname) %(refname)\" $prefix/r >actual-refs &&\n \ttest_cmp expected-refs actual-refs\n@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks update' '\n@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks create' '\n@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/foo $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks delete' '\n@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/foo $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect update' '\n@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect update' '\n@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect create' '\n@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect delete' '\n@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect no-deref update' '\n@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref update' '\n@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect no-deref create' '\n@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n@@ -463,7 +489,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success REFFILES 'non-empty directory blocks create' '\n-- \ngitgitgadget\n\n"},{"id":"430665","messageId":"7541a4b8d6dbdffbad06779e7c295a8e730ed9af.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 21/26] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:41Z","receivedAt":"2021-07-20T17:09:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex db7c0068fb5..4177274bea4 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"430666","messageId":"67f1282469d21a88eb253dced69e069d84f1be4f.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 18/26] reftable: implement stack, a mutable database of reftable files.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:38Z","receivedAt":"2021-07-20T17:09:52Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-stack.h | 128 ++++\n reftable/stack.c | 1396 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 947 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2514 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 06a5fb9103b..c18042929c8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2471,6 +2471,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 00000000000..1b602dda58a\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. After calling this,\n+ * reftable_addition_destroy should still be called.\n+ */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+/* print the entire stack represented by the directory */\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 00000000000..cf3b11ac998\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1396 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t// On Windows, can only unlink after closing.\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!rd) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i]) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t// On Windows, can only unlink after closing.\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (!names[i]) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len]) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (!add) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->time > 0 &&\n+\t\t log.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (!expiry && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (!dir) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_merged_table *merged = NULL;\n+\tstruct reftable_table table = { NULL };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\treftable_table_from_merged_table(&table, merged);\n+\terr = reftable_table_print(&table);\n+done:\n+\tif (stack)\n+\t\treftable_stack_destroy(stack);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 00000000000..f57005846e5\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 00000000000..0743defda13\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,947 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static char *get_tmp_dir(const char *prefix)\n+{\n+\tchar *dir = get_tmp_template(prefix);\n+\tEXPECT(mkdtemp(dir));\n+\treturn dir;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i]; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\tprintf(\"testing print functionality:\\n\");\n+\terr = reftable_stack_print_directory(dir, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_print_directory(dir, GIT_SHA256_FORMAT_ID);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\n+\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .new_hash = h1,\n+\t\t\t\t\t\t .old_hash = h2,\n+\t\t\t\t\t } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n+\n+\tinput.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\t/* even entries add the refs, odd entries delete them. */\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].update.new_hash =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].update.time = i;\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 100;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_reftable_stack_add);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c8db6852c35..996da85f7b5 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"430664","messageId":"3c9c3a2d56df2ef525072ede5cb420b2690bb979.1626800687.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH 22/26] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T17:04:42Z","receivedAt":"2021-07-20T17:09:53Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0031-reftable.sh | 6 ++++++\n 5 files changed, 14 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 19566c661f1..6014f74a1b8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2467,6 +2467,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 996da85f7b5..26b03d7b789 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 01201629fca..ed543037bb8 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -59,6 +59,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex cb90b7f4f7b..284cfe70d94 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nindex 7899a1c580e..f024968ed66 100755\n--- a/t/t0031-reftable.sh\n+++ b/t/t0031-reftable.sh\n@@ -288,4 +288,10 @@ test_expect_success 'FETCH_HEAD' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'dump reftable' '\n+\tinitialize &&\n+\thash_id=$(git config extensions.objectformat) &&\n+\ttest-tool dump-reftable $(test \"${hash_id}\" = \"sha256\" && echo \"-6\") -s .git/reftable\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"432839","messageId":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.git.git.1626800686.gitgitgadget@gmail.com","subject":"[PATCH v2 00/25] Support reftable ref backend for Git","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:51Z","receivedAt":"2021-08-16T20:17:27Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This continues the work in https://github.com/git/git/pull/847, which the\ngitgitgadget erroneously closed.\n\nChanges relative to last series (version 20 Jul 2021, tip: 4a5891f )\n\n * fix HTTP serving\n * fix t1501 (use absolute paths.)\n * fix git-new-workdir\n * fixes for FreeBSD (thanks, carenas@gmail.com!)\n\nFor the 'seen' branch, the following should be applied:\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d7137d1213..9323931eeb 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type);\n+ unsigned int *type, int *failure_errno);\n \n static void clear_reftable_log_record(struct reftable_log_record *log)\n {\n@@ -425,13 +424,14 @@ static int fixup_symrefs(struct ref_store *ref_store,\n for (i = 0; i < transaction->nr; i++) {\n struct ref_update *update = transaction->updates[i];\n struct object_id old_oid;\n+ int failure_errno;\n \n err = git_reftable_read_raw_ref(ref_store, update->refname,\n &old_oid, &referent,\n /* mutate input, like\n files-backend.c */\n- &update->type);\n- if (err < 0 && errno == ENOENT &&\n+ &update->type, &failure_errno);\n+ if (err < 0 && failure_errno == ENOENT &&\n is_null_oid(&update->old_oid)) {\n err = 0;\n }\n@@ -1602,7 +1603,7 @@ static int reftable_error_to_errno(int err)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type)\n+ unsigned int *type, int *failure_errno)\n {\n struct git_reftable_ref_store *refs =\n (struct git_reftable_ref_store *)ref_store;\n@@ -1626,13 +1627,11 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \n err = reftable_stack_read_ref(stack, refname, &ref);\n if (err > 0) {\n- errno = ENOENT;\n+ *failure_errno = ENOENT;\n err = -1;\n goto done;\n }\n if (err < 0) {\n- errno = reftable_error_to_errno(err);\n- err = -1;\n goto done;\n }\n\n\nHan-Wen Nienhuys (24):\n hash.h: provide constants for the hash IDs\n init-db: set the_repository->hash_algo early on\n reftable: RFC: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n Provide zlib's uncompress2 from compat/zlib-compat.c\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n refs: RFC: Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n t1301: document what needs to be done for reftable\n t1401,t2011: parameterize HEAD.lock for REFFILES\n t1404: annotate test cases with REFFILES\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 54 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 60 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +\n config.mak.uname | 3 +-\n configure.ac | 13 +\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n contrib/workdir/git-new-workdir | 2 +-\n hash.h | 6 +\n object-file.c | 7 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1686 +++++++++++++++++\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 +\n reftable/block.c | 448 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 107 ++\n reftable/error.c | 41 +\n reftable/generic.c | 169 ++\n reftable/generic.h | 32 +\n reftable/iter.c | 194 ++\n reftable/iter.h | 69 +\n reftable/merged.c | 362 ++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 292 +++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 801 ++++++++\n reftable/reader.h | 66 +\n reftable/readwrite_test.c | 651 +++++++\n reftable/record.c | 1200 ++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 408 ++++\n reftable/refname.c | 209 ++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 47 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 101 +\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 128 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 148 ++\n reftable/reftable.c | 115 ++\n reftable/stack.c | 1396 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 947 +++++++++\n reftable/system.h | 24 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 690 +++++++\n reftable/writer.h | 50 +\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 316 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1301-shared-repo.sh | 8 +-\n t/t1401-symbolic-ref.sh | 11 +-\n t/t1404-update-ref-errors.sh | 56 +-\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t2011-checkout-invalid-head.sh | 11 +-\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 94 files changed, 13156 insertions(+), 62 deletions(-)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/readwrite_test.c\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 5d213e46bb7b880238ff5ea3914e940a50ae9369\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1054%2Fhanwen%2Flibreftable-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1054/hanwen/libreftable-v2\nPull-Request: https://github.com/git/git/pull/1054\n\nRange-diff vs v1:\n\n 1: dc24dccf041 = 1: c40393e83e1 hash.h: provide constants for the hash IDs\n 2: f1167669848 = 2: 49dffeeae58 init-db: set the_repository->hash_algo early on\n 3: 76c2a1005da = 3: dba354eca12 reftable: RFC: add LICENSE\n 4: 616d6ed89ee = 4: 75864034af2 reftable: add error related functionality\n 5: 9d2cdfe3ddd = 5: eabb25d95b3 reftable: utility functions\n 6: 5ef4b7040a3 = 6: ad54d17f84b reftable: add blocksource, an abstraction for random access reads\n 7: 13a5cbef0df ! 7: 282d8e4d817 reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record_test.c (new)\n +\n +static void test_reftable_log_record_roundtrip(void)\n +{\n ++\tint i;\n +\tstruct reftable_log_record in[2] = {\n +\t\t{\n +\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n @@ reftable/record_test.c (new)\n +\t};\n +\tset_test_hash(in[0].update.new_hash, 1);\n +\tset_test_hash(in[0].update.old_hash, 2);\n -+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n ++\tfor (i = 0; i < ARRAY_SIZE(in); i++) {\n +\t\tstruct reftable_record rec = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n +\t\tuint8_t buffer[1024] = { 0 };\n 8: 27abd159658 ! 8: 893ee4ca21b Provide zlib's uncompress2 from compat/zlib-compat.c\n @@ Commit message\n \n This will be needed for reading reflog blocks in reftable.\n \n + Helped-by: Carlo Marcelo Arenas Belón \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n @@ Makefile: ifdef NO_DEFLATE_BOUND\n \n +ifdef NO_UNCOMPRESS2\n +\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n -+\tLIB_OBJS += compat/zlib-uncompress2.o\n ++\tREFTABLE_OBJS += compat/zlib-uncompress2.o\n +endif\n +\n ifdef NO_POSIX_GOODIES\n @@ compat/zlib-uncompress2.c (new)\n + err;\n +}\n \n + ## config.mak.uname ##\n +@@ config.mak.uname: ifeq ($(uname_S),FreeBSD)\n + \tFILENO_IS_A_MACRO = UnfortunatelyYes\n + endif\n + ifeq ($(uname_S),OpenBSD)\n ++\tNO_UNCOMPRESS2 = YesPlease\n + \tNO_STRCASESTR = YesPlease\n + \tNO_MEMMEM = YesPlease\n + \tUSE_ST_TIMESPEC = YesPlease\n +\n ## configure.ac ##\n @@ configure.ac: AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n 9: 25aa2bf9b71 ! 9: e4da0a2ef37 reftable: reading/writing blocks\n @@ reftable/block.c (new)\n +#include \n +\n +#ifdef NO_UNCOMPRESS2\n -+/* This is uncompress2, which is only available in zlib as of 2017.\n ++/*\n ++ * This is uncompress2, which is only available in zlib >= 1.2.9\n ++ * (released as of early 2017)\n + */\n +int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n +\t\tuLong *sourceLen);\n 10: 92970da9cb9 = 10: 0f325881389 reftable: a generic binary tree implementation\n 11: 3b817f37a01 = 11: 26cbc468f1f reftable: write reftable files\n 12: 99708f408b0 = 12: 69129fca086 reftable: generic interface to tables\n 13: fa9d3b95fd4 = 13: 22599d785b7 reftable: read reftable files\n 14: 3eeefa5665f ! 14: 45565bfb9c4 reftable: reftable file level tests\n @@ reftable/readwrite_test.c (new)\n +\t\t.block_size = 4096,\n +\t};\n +\tint err;\n ++\tint i;\n +\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n +\t\t\t\t\t .update_index = 0xa,\n +\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n @@ reftable/readwrite_test.c (new)\n +\t hash, to ensure that the compressed part is larger than the original.\n +\t*/\n +\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n -+\tfor (int i = 0; i < GIT_SHA1_RAWSZ; i++) {\n ++\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n +\t\thash1[i] = (uint8_t)(rand() % 256);\n +\t\thash2[i] = (uint8_t)(rand() % 256);\n +\t}\n 15: 4d7745e6552 = 15: 57b3e709c3e reftable: add a heap-based priority queue for reftable records\n 16: c906d79b341 = 16: f834cc5299f reftable: add merged table view\n 17: 806c2e04392 = 17: 15799d93ff7 reftable: implement refname validation\n 18: 67f1282469d = 18: b7728ceaef2 reftable: implement stack, a mutable database of reftable files.\n 19: 05e46f7e1d8 ! 19: 0f682be20bd reftable: add dump utility\n @@ Commit message\n inspecting a complete ref database\n \n Signed-off-by: Han-Wen Nienhuys \n + Helped-by: Carlo Marcelo Arenas Belón \n \n ## reftable/dump.c (new) ##\n @@\n @@ reftable/dump.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n ++#include \"git-compat-util.h\"\n ++#include \"hash.h\"\n ++\n +#include \"reftable-blocksource.h\"\n +#include \"reftable-error.h\"\n +#include \"reftable-merged.h\"\n @@ reftable/dump.c (new)\n +#include \"reftable-reader.h\"\n +#include \"reftable-stack.h\"\n +#include \"reftable-generic.h\"\n -+#include \"hash.h\"\n +\n +#include \n +#include \n 20: d1d4b2f9e9b ! 20: 754085752f4 refs: RFC: Reftable support for git-core\n @@ contrib/buildsystems/Generators/Vcxproj.pm: EOM\n print F << \"EOM\";\n \n \n + ## contrib/workdir/git-new-workdir ##\n +@@ contrib/workdir/git-new-workdir: trap cleanup $siglist\n + # create the links to the original repo. explicitly exclude index, HEAD and\n + # logs/HEAD from the list since they are purely related to the current working\n + # directory, and should not be shared.\n +-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n ++for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n + do\n + \t# create a containing directory if needed\n + \tcase $x in\n +\n ## refs.c ##\n @@\n #include \"repository.h\"\n @@ refs/reftable-backend.c (new)\n +\t\t.hash_id = the_hash_algo->format_id,\n +\t};\n +\tstruct strbuf sb = STRBUF_INIT;\n -+\tconst char *gitdir = path;\n ++\tstruct strbuf gitdir = STRBUF_INIT;\n +\tstruct strbuf wt_buf = STRBUF_INIT;\n +\tint wt = 0;\n +\n @@ refs/reftable-backend.c (new)\n +\t * backend, leading to infinite recursion. */\n +\twt = is_worktree(&wt_buf);\n +\tif (wt) {\n -+\t\tgitdir = wt_buf.buf;\n ++\t\tstrbuf_addbuf(&gitdir, &wt_buf);\n ++\t} else {\n ++\t\tstrbuf_realpath(&gitdir, path, /*die_on_error=*/0);\n +\t}\n +\n +\tbase_ref_store_init(ref_store, &refs_be_reftable);\n -+\tref_store->gitdir = xstrdup(gitdir);\n ++\tref_store->gitdir = xstrdup(gitdir.buf);\n +\trefs->store_flags = store_flags;\n -+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n ++\tstrbuf_addf(&sb, \"%s/reftable\", gitdir.buf);\n +\trefs->reftable_dir = xstrdup(sb.buf);\n +\tstrbuf_reset(&sb);\n +\n @@ refs/reftable-backend.c (new)\n +\n +\tstrbuf_release(&sb);\n +\tstrbuf_release(&wt_buf);\n ++\tstrbuf_release(&gitdir);\n +\treturn ref_store;\n +}\n +\n @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)\n \t\t}\n \t}\n \t/*\n +@@ setup.c: void check_repository_format(struct repository_format *fmt)\n + \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n + \tthe_repository->repository_format_partial_clone =\n + \t\txstrdup_or_null(fmt->partial_clone);\n ++\t/* XXX why is repo->ref_storage_format set in multiple places?! */\n ++\tthe_repository->ref_storage_format = xstrdup_or_null(fmt->ref_storage);\n + \tclear_repository_format(&repo_fmt);\n + }\n + \n \n ## t/t0031-reftable.sh (new) ##\n @@\n @@ t/t0031-reftable.sh (new)\n +\ttest_cmp expect actual\n +'\n +\n ++. \"$TEST_DIRECTORY\"/lib-httpd.sh\n ++start_httpd\n ++\n ++REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n ++\n ++test_expect_success 'serving ls-remote' '\n ++\tGIT_TEST_REFTABLE=1 git init -b main \"$REPO\" &&\n ++\tcd \"$REPO\" &&\n ++\ttest_commit m1 &&\n ++\t>.git/git-daemon-export-ok &&\n ++\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n ++\tcat << EOF >expect &&\n ++HEAD\n ++refs/heads/main\n ++refs/tags/m1\n ++EOF\n ++\ttest_cmp actual expect\n ++'\n ++\n +test_done\n \n ## t/t1409-avoid-packing-refs.sh ##\n 21: 7541a4b8d6d = 21: 79c4b90d0f4 git-prompt: prepare for reftable refs backend\n 22: 3c9c3a2d56d ! 22: 1f48091e549 Add \"test-tool dump-reftable\" command.\n @@ t/t0031-reftable.sh: test_expect_success 'FETCH_HEAD' '\n +\ttest-tool dump-reftable $(test \"${hash_id}\" = \"sha256\" && echo \"-6\") -s .git/reftable\n +'\n +\n - test_done\n + . \"$TEST_DIRECTORY\"/lib-httpd.sh\n + start_httpd\n + \n 23: 73eece0caac = 23: 64dc87746db t1301: document what needs to be done for reftable\n 24: d895736f0e1 = 24: c248b69c6e3 t1401,t2011: parameterize HEAD.lock for REFFILES\n 25: f0a528b9ece = 25: 9f1b792459e t1404: annotate test cases with REFFILES\n 26: 4a5891fa8cc < -: ----------- t7004: avoid direct filesystem access\n\n-- \ngitgitgadget\n"},{"id":"432840","messageId":"c40393e83e1d57be9429211cdebcb715e9d319d8.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 01/25] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:52Z","receivedAt":"2021-08-16T20:17:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will simplify referencing them from code that is not deeply integrated with\nGit, in particular, the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n hash.h | 6 ++++++\n object-file.c | 7 ++-----\n 2 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 27a180248f9..f9e8d4b736c 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -95,12 +95,18 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n /* Number of algorithms supported (including unknown). */\n #define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n+/* \"sha1\", big-endian */\n+#define GIT_SHA1_FORMAT_ID 0x73686131\n+\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* \"s256\", big-endian */\n+#define GIT_SHA256_FORMAT_ID 0x73323536\n+\n /* The length in bytes and in hex digits of an object name (SHA-256 value). */\n #define GIT_SHA256_RAWSZ 32\n #define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\ndiff --git a/object-file.c b/object-file.c\nindex a8be8994814..7bfd5e6e2e9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -164,7 +164,6 @@ static void git_hash_unknown_final_oid(struct object_id *oid, git_hash_ctx *ctx)\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n-\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\tNULL,\n@@ -183,8 +182,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha1\",\n-\t\t/* \"sha1\", big-endian */\n-\t\t0x73686131,\n+\t\tGIT_SHA1_FORMAT_ID,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n \t\tGIT_SHA1_BLKSZ,\n@@ -199,8 +197,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha256\",\n-\t\t/* \"s256\", big-endian */\n-\t\t0x73323536,\n+\t\tGIT_SHA256_FORMAT_ID,\n \t\tGIT_SHA256_RAWSZ,\n \t\tGIT_SHA256_HEXSZ,\n \t\tGIT_SHA256_BLKSZ,\n-- \ngitgitgadget\n\n"},{"id":"432841","messageId":"49dffeeae5866d01e8a88b21e722bf047e2eb778.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 02/25] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:53Z","receivedAt":"2021-08-16T20:17:37Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 2167796ff2a..c2f03f6018e 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -425,6 +425,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"432842","messageId":"dba354eca12a53b4d76d0d7475a4128843c7f70d.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 03/25] reftable: RFC: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:54Z","receivedAt":"2021-08-16T20:17:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe objective of this code is to be usable as a C library, so it can be reused\nin libgit2.\n\nThis is currently using a BSD license as it is the liberal license I could find,\nbut this could be changed to whatever fits the stated goal above.\n\nThis code is currently imported from github.com/hanwen/reftable. Once this code\nlands in git.git, the C code will be removed from github.com/hanwen/reftable,\nand the git.git code will be the source of truth.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 00000000000..402e0f9356b\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"432843","messageId":"75864034af2b2671697cad92f10a3335dc841b1e.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 04/25] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:55Z","receivedAt":"2021-08-16T20:17:39Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 00000000000..f6f16def921\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 00000000000..6f89bedf1a5\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432845","messageId":"eabb25d95b37f90c1b0f48764b62ebd1f802fff2.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 05/25] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:56Z","receivedAt":"2021-08-16T20:17:40Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 24 ++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 545 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d7..addf2813086 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -743,6 +743,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -821,6 +822,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1195,7 +1198,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2442,7 +2445,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2458,6 +2469,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2608,6 +2621,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2895,7 +2914,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3225,7 +3244,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 171b4124afe..c2bf5bdffc6 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -640,6 +640,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -662,7 +668,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -902,11 +908,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 00000000000..f761e48028c\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (!a) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 00000000000..096b36862b9\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 00000000000..1fcd2297256\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(!out[2]);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!out[1]);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 00000000000..bd0a02d3f68\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 00000000000..5f2185f1f34\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 00000000000..5e7698ae654\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 00000000000..bf963ee458e\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,24 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+// This header glues the reftable library to the rest of Git\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hash.h\" /* hash ID, sizes.*/\n+#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+\n+#include \n+\n+struct strbuf;\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 00000000000..84ac972cad0\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_FORMAT_ID));\n+}\n+\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add(b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 00000000000..774cb275bf6\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 00000000000..3b58e423e7b\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53a..f7c888ffda7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -53,13 +53,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f5228508..25f77469146 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -49,6 +49,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 00000000000..0ed14971a58\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"432844","messageId":"ad54d17f84bdb581069d57dac3796de7b81b8e63.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 06/25] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:57Z","receivedAt":"2021-08-16T20:17:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex addf2813086..86ba58cf140 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2447,6 +2447,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 00000000000..0044eecd9aa\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(!bs->ops);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(!bs->ops);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 00000000000..072e2727ad2\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 00000000000..5aa3990a573\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432847","messageId":"893ee4ca21bc75910f88bd8ae586fde02795e2cd.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 08/25] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:59Z","receivedAt":"2021-08-16T20:17:44Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be needed for reading reflog blocks in reftable.\n\nHelped-by: Carlo Marcelo Arenas Belón \nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 7 +++\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n config.mak.uname | 1 +\n configure.ac | 13 ++++++\n 6 files changed, 115 insertions(+)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n\ndiff --git a/Makefile b/Makefile\nindex e98d8ed17cf..16c883978d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -256,6 +256,8 @@ all::\n #\n # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n #\n+# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n+#\n # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n #\n@@ -1738,6 +1740,11 @@ ifdef NO_DEFLATE_BOUND\n \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n endif\n \n+ifdef NO_UNCOMPRESS2\n+\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n+\tREFTABLE_OBJS += compat/zlib-uncompress2.o\n+endif\n+\n ifdef NO_POSIX_GOODIES\n \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n endif\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..5711c63979d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -224,6 +224,7 @@ linux-gcc-default)\n \t;;\n Linux32)\n \tCC=gcc\n+\tMAKEFLAGS=\"$MAKEFLAGS NO_UNCOMPRESS2=1\"\n \t;;\n linux-musl)\n \tCC=gcc\ndiff --git a/compat/.gitattributes b/compat/.gitattributes\nnew file mode 100644\nindex 00000000000..40dbfb170da\n--- /dev/null\n+++ b/compat/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\nnew file mode 100644\nindex 00000000000..6893bb469ce\n--- /dev/null\n+++ b/compat/zlib-uncompress2.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress2 (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 69413fb3dc0..61e11550b1f 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -256,6 +256,7 @@ ifeq ($(uname_S),FreeBSD)\n \tFILENO_IS_A_MACRO = UnfortunatelyYes\n endif\n ifeq ($(uname_S),OpenBSD)\n+\tNO_UNCOMPRESS2 = YesPlease\n \tNO_STRCASESTR = YesPlease\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\ndiff --git a/configure.ac b/configure.ac\nindex 031e8d3fee8..c3a913103d0 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n LIBS=\"$old_LIBS\"\n \n+AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n+AC_LANG_PROGRAM([#include ],\n+ [uncompress2(NULL,NULL,NULL,NULL);])])\n+AC_MSG_CHECKING([for uncompress2 in -lz])\n+old_LIBS=\"$LIBS\"\n+LIBS=\"$LIBS -lz\"\n+AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n+\t[AC_MSG_RESULT([yes])],\n+\t[AC_MSG_RESULT([no])\n+\tNO_UNCOMPRESS2=yes])\n+LIBS=\"$old_LIBS\"\n+\n GIT_UNSTASH_FLAGS($ZLIB_PATH)\n \n GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n+GIT_CONF_SUBST([NO_UNCOMPRESS2])\n \n #\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n-- \ngitgitgadget\n\n"},{"id":"432846","messageId":"282d8e4d817f969a3c4d882cc833dd76f4451764.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 07/25] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:16:58Z","receivedAt":"2021-08-16T20:17:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1200 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 408 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1885 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 86ba58cf140..e98d8ed17cf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2449,7 +2449,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 00000000000..5eee72c4c11\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 00000000000..34ed480b257\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1200 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = rec;\n+\tconst struct reftable_ref_record *src = src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release(rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tobj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));\n+\tCOPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->update.name,\n+\t\t log->update.email, log->update.time,\n+\t\t log->update.tz_offset);\n+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->update.email) {\n+\t\t\tdst->update.email = xstrdup(dst->update.email);\n+\t\t}\n+\t\tif (dst->update.name) {\n+\t\t\tdst->update.name = xstrdup(dst->update.name);\n+\t\t}\n+\t\tif (dst->update.message) {\n+\t\t\tdst->update.message = xstrdup(dst->update.message);\n+\t\t}\n+\n+\t\tif (dst->update.new_hash) {\n+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tif (dst->update.old_hash) {\n+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->update.new_hash);\n+\t\treftable_free(r->update.old_hash);\n+\t\treftable_free(r->update.name);\n+\t\treftable_free(r->update.email);\n+\t\treftable_free(r->update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_log_record *r = rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->update.old_hash;\n+\tnewh = r->update.new_hash;\n+\tif (!oldh) {\n+\t\toldh = zero;\n+\t}\n+\tif (!newh) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->update.old_hash);\n+\t\t\tFREE_AND_NULL(r->update.new_hash);\n+\t\t\tFREE_AND_NULL(r->update.message);\n+\t\t\tFREE_AND_NULL(r->update.email);\n+\t\t\tFREE_AND_NULL(r->update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n+\n+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n+\tmemcpy(r->update.name, dest.buf, dest.len);\n+\tr->update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n+\tmemcpy(r->update.email, dest.buf, dest.len);\n+\tr->update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n+\tmemcpy(r->update.message, dest.buf, dest.len);\n+\tr->update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (!a)\n+\t\ta = empty;\n+\n+\tif (!b)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (!a)\n+\t\ta = zero;\n+\n+\tif (!b)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->update.name, b->update.name) &&\n+\t\t a->update.time == b->update.time &&\n+\t\t a->update.tz_offset == b->update.tz_offset &&\n+\t\t null_streq(a->update.email, b->update.email) &&\n+\t\t null_streq(a->update.message, b->update.message) &&\n+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n+\t\t\t\t hash_size) &&\n+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n+\t\t\t\t hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_index_record *rec = r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = rec;\n+\tconst struct reftable_index_record *src = src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a && b)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tconst struct reftable_log_record *la = a;\n+\tconst struct reftable_log_record *lb = b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 00000000000..498e8c50bf4\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 00000000000..d6e5bbfdd61\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,408 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tint i;\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t.time = 1577123507,\n+\t\t\t\t.tz_offset = 100,\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].update.new_hash, 1);\n+\tset_test_hash(in[0].update.old_hash, 2);\n+\tfor (i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 00000000000..7985b94ae2c\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t};\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b..09d4b83ef9b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432849","messageId":"e4da0a2ef370c5559543294e8b4e6142613e6bf2.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 09/25] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:00Z","receivedAt":"2021-08-16T20:17:47Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/block.c | 448 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 699 insertions(+)\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 16c883978d4..1d396c87be9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,10 +2454,12 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 00000000000..eb5268dd3a6\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,448 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \n+\n+#ifdef NO_UNCOMPRESS2\n+/*\n+ * This is uncompress2, which is only available in zlib >= 1.2.9\n+ * (released as of early 2017)\n+ */\n+int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n+\t\tuLong *sourceLen);\n+#endif\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK !=\n+\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n+\t\t\t\tblock->data + block_header_skip, &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp && source.ops)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 00000000000..e207706a644\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 00000000000..c3d35eedb98\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b..c9deeaf08c7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432848","messageId":"0f325881389b3631ce03ce8ff1ee8bad44096a8e.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 10/25] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:01Z","receivedAt":"2021-08-16T20:17:49Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 1d396c87be9..be14b66f9e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,11 +2458,13 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 00000000000..82db7995dd6\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 00000000000..fbdd002e23a\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 00000000000..09a970e17b9\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = arg;\n+\tif (c->last) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7..050551fa698 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432850","messageId":"26cbc468f1f1ed86219cdef7ad549736a4748b1c.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 11/25] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:02Z","receivedAt":"2021-08-16T20:17:49Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 148 ++++++++\n reftable/writer.c | 690 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex be14b66f9e3..9460f7de974 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2459,6 +2459,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 00000000000..af36462ced5\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \"reftable-record.h\"\n+\n+#include \n+#include /* ssize_t */\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 00000000000..1baad069b64\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,690 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\t\t\t 1 :\n+\t\t\t 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy(dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = -1;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->update.message;\n+\tif (!w->opts.exact_log_message && log->update.message) {\n+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tif (arg->last) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 00000000000..09b88673d97\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tssize_t (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432851","messageId":"69129fca086a303701f36412c61d8a5bb5e977df.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 12/25] reftable: generic interface to tables","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:03Z","receivedAt":"2021-08-16T20:17:59Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/generic.c | 169 +++++++++++++++++++++++++++++++++++\n reftable/generic.h | 32 +++++++\n reftable/reftable-generic.h | 47 ++++++++++\n reftable/reftable-iterator.h | 39 ++++++++\n reftable/reftable.c | 115 ++++++++++++++++++++++++\n 6 files changed, 405 insertions(+)\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex 9460f7de974..f45a981bc20 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,9 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n+REFTABLE_OBJS += reftable/generic.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/generic.c b/reftable/generic.c\nnew file mode 100644\nindex 00000000000..7a8a738d860\n--- /dev/null\n+++ b/reftable/generic.c\n@@ -0,0 +1,169 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = ~((uint64_t)0),\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_print(struct reftable_table *tab) {\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tuint32_t hash_id = reftable_table_hash_id(tab);\n+\tint err = reftable_table_seek_ref(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_table_seek_log(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn 0;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 00000000000..98886a06402\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 00000000000..d239751a778\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,47 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* dump table contents onto stdout for debugging */\n+int reftable_table_print(struct reftable_table *tab);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 00000000000..d3eee7af357\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 00000000000..0e4607a7cd6\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \ngitgitgadget\n\n"},{"id":"432852","messageId":"22599d785b7a5893912ad1c4848e9ef44b6180dc.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 13/25] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:04Z","receivedAt":"2021-08-16T20:17:59Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 194 +++++++++\n reftable/iter.h | 69 ++++\n reftable/reader.c | 801 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 +++\n reftable/reftable-reader.h | 101 +++++\n 6 files changed, 1233 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex f45a981bc20..5c857989091 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2456,7 +2456,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/generic.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 00000000000..93d04f735b8\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,194 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn !it->ops;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\treftable_free(it->offsets);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 00000000000..09eb0cbfa59\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,69 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+\n+/* Takes ownership of `offsets` */\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 00000000000..49f4ec070e6\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,801 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (!source->ops) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\tuint64_t file_size = block_source_size(source);\n+\n+\t/* Need +1 to read type of first block. */\n+\tuint32_t read_size = header_size(2) + 1; /* read v2 because it's larger. */\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\tif (read_size > file_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = block_source_read_block(source, &header, 0, read_size);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = file_size - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (!ti->bi.br) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next(ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek(tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id(tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+\n+int reftable_reader_print_file(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_reader *r = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\treftable_table_from_reader(&tab, r);\n+\terr = reftable_table_print(&tab);\n+done:\n+\treftable_reader_free(r);\n+\treturn err;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 00000000000..39583e5dbcd\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 00000000000..4a4bc2fdf85\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,101 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* print table onto stdout for debugging. */\n+int reftable_reader_print_file(const char *tablename);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432853","messageId":"45565bfb9c4bafd4579a78833653907ad42387d2.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 14/25] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:05Z","receivedAt":"2021-08-16T20:18:01Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/readwrite_test.c | 651 ++++++++++++++++++++++++++++++++++++++\n reftable/reftable-tests.h | 2 +-\n t/helper/test-reftable.c | 1 +\n 4 files changed, 654 insertions(+), 1 deletion(-)\n create mode 100644 reftable/readwrite_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5c857989091..5e2ff446894 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2469,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nnew file mode 100644\nindex 00000000000..8ed817f4609\n--- /dev/null\n+++ b/reftable/readwrite_test.c\n@@ -0,0 +1,651 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.new_hash = hash;\n+\t\tlog.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == GIT_SHA256_FORMAT_ID) ? 2 :\n+\t\t\t\t\t\t\t\t\t\t 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n+\t\t\t\t\t .update_index = 0xa,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t\t } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.update.old_hash = hash1;\n+\tlog.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.old_hash = hash1;\n+\t\tlog.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n+\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_write_empty_table(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_write_key_order(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record refs[2] = {\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}, {\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}\n+\t};\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\terr = reftable_writer_add_ref(w, &refs[0]);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_add_ref(w, &refs[1]);\n+\tprintf(\"%d\\n\", err);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_writer_close(w);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_corrupt_table_empty(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+}\n+\n+static void test_corrupt_table(void)\n+{\n+\tuint8_t zeros[1024] = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\tstrbuf_add(&buf, zeros, sizeof(zeros));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\tstrbuf_release(&buf);\n+}\n+\n+int readwrite_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_corrupt_table);\n+\tRUN_TEST(test_corrupt_table_empty);\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_write_key_order);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_write_empty_table);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654..3d541fa5c0c 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n-int reftable_test_main(int argc, const char **argv);\n+int readwrite_test_main(int argc, const char **argv);\n int stack_test_main(int argc, const char **argv);\n int tree_test_main(int argc, const char **argv);\n int reftable_dump_main(int argc, char *const *argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa698..898aba836fd 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432854","messageId":"57b3e709c3e32f7dbcbe9a5d8a14f4d9bd9f47a1.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 15/25] reftable: add a heap-based priority queue for reftable records","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:06Z","receivedAt":"2021-08-16T20:18:01Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5e2ff446894..744a63b9d2c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n@@ -2468,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 00000000000..8918d158e2d\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 00000000000..385d2fb139a\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 00000000000..ad21673e854\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 3d541fa5c0c..0019cbcfa49 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ https://developers.google.com/open-source/licenses/bsd\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int readwrite_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 898aba836fd..0b5a1701df1 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"432855","messageId":"f834cc5299f1bda86561a679c4d05c237381200e.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 16/25] reftable: add merged table view","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:07Z","receivedAt":"2021-08-16T20:18:02Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 764 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 744a63b9d2c..defcc054fb6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2469,6 +2470,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 00000000000..e5b53da6db3\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,362 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (!mt) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record(tab, it, rec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 00000000000..8c4d4d58d77\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 00000000000..1e2afe37b8b\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,292 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n+\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == GIT_SHA1_FORMAT_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 00000000000..1a6d16915ab\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 0b5a1701df1..8087f2da4e6 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"432856","messageId":"15799d93ff7713cf9f9564811de817b788a7f142.1629145036.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 17/25] reftable: implement refname validation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:08Z","receivedAt":"2021-08-16T20:18:04Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 342 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex defcc054fb6..04b143f670d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2474,6 +2474,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 00000000000..95734969324\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 00000000000..a24b40fcb42\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 00000000000..8645cd93bbd\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 8087f2da4e6..c8db6852c35 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"432859","messageId":"b7728ceaef2658b7bb520a4f436a0acab6807b23.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 18/25] reftable: implement stack, a mutable database of reftable files.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:09Z","receivedAt":"2021-08-16T20:18:05Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-stack.h | 128 ++++\n reftable/stack.c | 1396 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 947 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2514 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 04b143f670d..152bbd6c134 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2475,6 +2475,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 00000000000..1b602dda58a\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. After calling this,\n+ * reftable_addition_destroy should still be called.\n+ */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+/* print the entire stack represented by the directory */\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 00000000000..cf3b11ac998\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1396 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t// On Windows, can only unlink after closing.\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!rd) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i]) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t// On Windows, can only unlink after closing.\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (!names[i]) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len]) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (!add) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->time > 0 &&\n+\t\t log.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (!expiry && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (!dir) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_merged_table *merged = NULL;\n+\tstruct reftable_table table = { NULL };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\treftable_table_from_merged_table(&table, merged);\n+\terr = reftable_table_print(&table);\n+done:\n+\tif (stack)\n+\t\treftable_stack_destroy(stack);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 00000000000..f57005846e5\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 00000000000..0743defda13\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,947 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static char *get_tmp_dir(const char *prefix)\n+{\n+\tchar *dir = get_tmp_template(prefix);\n+\tEXPECT(mkdtemp(dir));\n+\treturn dir;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i]; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\tprintf(\"testing print functionality:\\n\");\n+\terr = reftable_stack_print_directory(dir, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_print_directory(dir, GIT_SHA256_FORMAT_ID);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\n+\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .new_hash = h1,\n+\t\t\t\t\t\t .old_hash = h2,\n+\t\t\t\t\t } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n+\n+\tinput.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\t/* even entries add the refs, odd entries delete them. */\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].update.new_hash =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].update.time = i;\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 100;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_reftable_stack_add);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c8db6852c35..996da85f7b5 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432857","messageId":"0f682be20bd360e68118a4e5dfa4ae817ad7cc50.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 19/25] reftable: add dump utility","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:10Z","receivedAt":"2021-08-16T20:18:06Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Carlo Marcelo Arenas Belón \n---\n reftable/dump.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 107 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 00000000000..155953d1b82\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,107 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"git-compat-util.h\"\n+#include \"hash.h\"\n+\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n+#include \"reftable-generic.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -6 sha256 hash format\\n\"\n+\t \" -h this help\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n+\t\t\tbreak;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n+\t\t\topt_dump_table = 1;\n+\t\telse if (!strcmp(\"-6\", argv[1]))\n+\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n+\t\t\topt_dump_stack = 1;\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n+\t\t\topt_compact = 1;\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t}\n+\n+\tif (argc != 2) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[1];\n+\n+\tif (opt_dump_table) {\n+\t\terr = reftable_reader_print_file(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = reftable_stack_print_directory(arg, opt_hash_id);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \ngitgitgadget\n\n"},{"id":"432858","messageId":"79c4b90d0f47404d11f94dc1dc946b14b0018b7f.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 21/25] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:12Z","receivedAt":"2021-08-16T20:18:07Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex db7c0068fb5..4177274bea4 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"432862","messageId":"754085752f49864a52d502986b8491ba0db2e2d4.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 20/25] refs: RFC: Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:11Z","receivedAt":"2021-08-16T20:18:08Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see Documentation/technical/reftable.txt.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by setting GIT_TEST_REFTABLE in the environment. When\nGIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set.\n\nThere is no option to git-init for now, as the test suite still shows failures\nwith GIT_TEST_REFTABLE=1.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nHelped-by: Patrick Steinhardt \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 39 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1686 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/t0031-reftable.sh | 310 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 23 files changed, 2154 insertions(+), 29 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdca..82c5940f143 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffd..72576235833 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex 152bbd6c134..6d0074939e5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -990,6 +990,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66fe66679c8..baa1ff4fc60 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c2f03f6018e..22b07d2b2fb 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -226,6 +228,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -235,6 +238,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -249,9 +270,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -268,7 +286,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -383,7 +401,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -422,6 +440,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -476,6 +495,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -549,6 +571,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -713,5 +736,5 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..d43d6898039 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n static int do_clear_stash(void)\n {\n \tstruct object_id obj;\n+\tint result;\n \tif (get_oid(ref_stash, &obj))\n \t\treturn 0;\n \n-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n+\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n+\n+\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n+\t * even when refs are deleted. */\n+\tdelete_reflog(ref_stash);\n+\treturn result;\n }\n \n static int clear_stash(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 0d0a80da61f..6f4facebc61 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -328,9 +329,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex bd4869beee4..777d99013dd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -656,9 +656,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1076,6 +1077,7 @@ struct repository_format {\n \tint hash_algo;\n \tint sparse_index;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 61e11550b1f..019c88d5df4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -712,7 +712,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba1..1a25789d285 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ EOM\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ EOM\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a5215..989197aace0 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/refs.c b/refs.c\nindex 8b9f7c3a80a..00c4d24df6e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,15 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1880,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1902,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1962,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1976,7 @@ done:\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1990,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0..5a6d4ca9fa8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 3155708345f..e36f215067e 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -672,6 +672,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..52fe6bc7a4b\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1686 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * two separate databases (ie. two reftable/ directories), one for the\n+ * repository, and one for the worktree. For reading, we merge the view (see\n+ * git_reftable_iterator) of both, when necessary.\n+ *\n+ * Unfortunately, the worktrees can also be selected by specifying a magic\n+ * refname (eg. worktree/BLA/refname, even if BLA isn't the current worktree.)\n+ */\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tconst char *wtname = refname;\n+\tint wtname_len = 0;\n+\tconst char *wtref = refname;\n+\n+\tif (refname == NULL)\n+\t\treturn store->main_stack;\n+\n+\tif (!parse_worktree_ref(refname, &wtname, &wtname_len, &wtref) &&\n+\t wtname_len) {\n+\t\t/* this makes me cry. Woe you if you try to access\n+\t\t * worktree/BLA/REF and the current worktree\n+\t\t * from the same process.\n+\t\t */\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_write_options cfg = {\n+\t\t\t.block_size = 4096,\n+\t\t\t.hash_id = the_hash_algo->format_id,\n+\t\t};\n+\n+\t\tstrbuf_addstr(&wt_dir, store->base.gitdir);\n+\t\tstrbuf_addstr(&wt_dir, \"/worktrees/\");\n+\t\tstrbuf_add(&wt_dir, wtname, wtname_len);\n+\t\tstrbuf_addstr(&wt_dir, \"/reftable\");\n+\n+\t\tif (store->worktree_stack)\n+\t\t\treftable_stack_destroy(store->worktree_stack);\n+\t\tstore->err = reftable_new_stack(&store->worktree_stack,\n+\t\t\t\t\t\twt_dir.buf, cfg);\n+\t\tassert(store->err != REFTABLE_API_ERROR);\n+\n+\t\treturn store->worktree_stack;\n+\t}\n+\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *out = ref;\n+\tint name_len = 0;\n+\tif (skip_prefix(ref, \"main-worktree/\", &out))\n+\t\treturn out;\n+\n+\tif (!parse_worktree_ref(ref, NULL, &name_len, &out) && name_len) {\n+\t\treturn out;\n+\t}\n+\n+\treturn ref;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->update.old_hash = NULL;\n+\t\tlog->update.new_hash = NULL;\n+\t\tlog->update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path. Mutates its arg, trimming it to the\n+ * gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf gitdir = STRBUF_INIT;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_realpath(&wt_buf, path, /*die_on_error=*/0);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tstrbuf_addbuf(&gitdir, &wt_buf);\n+\t} else {\n+\t\tstrbuf_realpath(&gitdir, path, /*die_on_error=*/0);\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir.buf);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir.buf);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\n+\t\trefs->err =\n+\t\t\treftable_new_stack(&refs->worktree_stack, sb.buf, cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\tstrbuf_release(&gitdir);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs,\n+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_id = *null_oid();\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->update.new_hash = u->new_oid.hash;\n+\t\tlog->update.message = u->msg;\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\n+\t\tlog->update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.update.message = xstrdup(arg->logmsg);\n+\t\tlog.update.new_hash = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.update.old_hash != NULL ||\n+\t\t log.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.update.message = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tlog.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\told_ref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].update.message = (char *)arg->logmsg;\n+\t\ttodo[0].update.old_hash = val1;\n+\t\ttodo[0].update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].update.old_hash = NULL;\n+\t\ttodo[1].update.new_hash = val1;\n+\t\ttodo[1].update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* this copies the entire reflog history. Is this the right semantics?\n+\t */\n+\t/* XXX should clear out existing reflog entries for oldname? */\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\toidread(&ri->oid, ri->log.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.update.old_hash);\n+\t\toidread(&new_oid, log.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->update.old_hash);\n+\t\toidread(&new_oid, log->update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].update.old_hash);\n+\t\toidread(&noid, logs[i].update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].update.email,\n+\t\t\t\t (timestamp_t)logs[i].update.time,\n+\t\t\t\t logs[i].update.tz_offset,\n+\t\t\t\t logs[i].update.message, policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash != NULL) {\n+\t\t\t\tdest->update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* XXX - skip writing records that were not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t} else {\n+\t\t/* XXX - print something */\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex b2bf44c6faf..f1cc8df47c4 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -180,6 +180,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex 3740c93bc0f..1bd9b4d09c8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -82,6 +82,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex eb9367ca5cb..58504dfeb68 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -498,6 +498,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -648,6 +651,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1312,6 +1316,8 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\tthe_repository->repository_format_partial_clone =\n \t\t\t\trepo_fmt.partial_clone;\n \t\t\trepo_fmt.partial_clone = NULL;\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n \t\t}\n \t}\n \t/*\n@@ -1399,6 +1405,8 @@ void check_repository_format(struct repository_format *fmt)\n \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tthe_repository->repository_format_partial_clone =\n \t\txstrdup_or_null(fmt->partial_clone);\n+\t/* XXX why is repo->ref_storage_format set in multiple places?! */\n+\tthe_repository->ref_storage_format = xstrdup_or_null(fmt->ref_storage);\n \tclear_repository_format(&repo_fmt);\n }\n \ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 00000000000..d2b398ba5dc\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,310 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+write_script fake_editor <<\\EOF\n+echo \"$MSG\" >\"$1\"\n+echo \"$MSG\" >&2\n+EOF\n+GIT_EDITOR=./fake_editor\n+export GIT_EDITOR\n+\n+\n+test_expect_success 'using reftable' '\n+\tinitialize &&\n+\ttest -d .git/reftable &&\n+\ttest -f .git/reftable/tables.list\n+'\n+\n+test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n+\tinitialize &&\n+\ttest_commit 1st &&\n+\ttest_commit 2nd &&\n+\tMSG=b4 git notes add &&\n+\tMSG=b3 git notes edit &&\n+\techo b4 >expect &&\n+\tgit notes --ref commits@{1} show >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git reflog delete' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_commit file2 &&\n+\ttest_commit file3 &&\n+\ttest_commit file4 &&\n+\tgit reflog delete HEAD@{1} &&\n+\tgit reflog > output &&\n+\t! grep file3 output\n+'\n+\n+test_expect_success 'branch -D delete nonexistent branch' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_must_fail git branch -D ../../my-private-file\n+'\n+\n+test_expect_success 'branch copy' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\ttest_commit file2 &&\n+\tgit branch src &&\n+\tgit reflog src > expect &&\n+\tgit branch -c src dst &&\n+\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'update-ref on corrupted data' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tOLD_SHA1=$(git rev-parse HEAD) &&\n+\ttest_commit file2 &&\n+\tls -l .git/reftable &&\n+\tfor f in .git/reftable/*.ref\n+\tdo\n+\t\t>$f\n+\tdone &&\n+\ttest_must_fail git update-ref refs/heads/main $OLD_SHA1\n+'\n+\n+test_expect_success 'git stash' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttouch actual expected &&\n+\tgit -c status.showStash=true status >expected &&\n+\techo hoi >> file.t &&\n+\tgit stash push -m stashed &&\n+\tgit stash clear &&\n+\tgit -c status.showStash=true status >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'rename branch' '\n+\tinitialize &&\n+\tgit symbolic-ref HEAD refs/heads/before &&\n+\ttest_commit file &&\n+\tgit show-ref | sed s/before/after/g > expected &&\n+\tgit branch -M after &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(GIT_TEST_REFTABLE=1 git_init || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\t(GIT_TEST_REFTABLE=1 git_init start) &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tGIT_TEST_REFTABLE=1 git init -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat << EOF >expect &&\n+HEAD\n+refs/heads/main\n+refs/tags/m1\n+EOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506..cdc21bf2dcb 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; need files backend'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..6bdd430dfe3 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 577f32dc71f..e523c3dd624 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; requires files ref backend'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..375e1dfd663 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1524,7 +1524,12 @@ parisc* | hppa*)\n \t;;\n esac\n \n-test_set_prereq REFFILES\n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq !REFFILES\n+else\n+ test_set_prereq REFFILES\n+fi\n \n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n-- \ngitgitgadget\n\n"},{"id":"432860","messageId":"1f48091e549d2d6b8c1e0da2c232624ac0fd47b3.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 22/25] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:13Z","receivedAt":"2021-08-16T20:18:10Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0031-reftable.sh | 6 ++++++\n 5 files changed, 14 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 6d0074939e5..98b8f1a1f9f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2471,6 +2471,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 996da85f7b5..26b03d7b789 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex f7c888ffda7..338a57b104d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -61,6 +61,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 25f77469146..48cee1f4a2d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nindex d2b398ba5dc..c76e9042e1d 100755\n--- a/t/t0031-reftable.sh\n+++ b/t/t0031-reftable.sh\n@@ -288,6 +288,12 @@ test_expect_success 'FETCH_HEAD' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'dump reftable' '\n+\tinitialize &&\n+\thash_id=$(git config extensions.objectformat) &&\n+\ttest-tool dump-reftable $(test \"${hash_id}\" = \"sha256\" && echo \"-6\") -s .git/reftable\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \ngitgitgadget\n\n"},{"id":"432861","messageId":"64dc87746db87b31e741cfa95e9da0fc7836df37.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 23/25] t1301: document what needs to be done for reftable","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:14Z","receivedAt":"2021-08-16T20:18:12Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1301-shared-repo.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh\nindex 84bf1970d8b..a5755b4a434 100755\n--- a/t/t1301-shared-repo.sh\n+++ b/t/t1301-shared-repo.sh\n@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' '\n \t)\n '\n \n+# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/\n for u in 002 022\n do\n-\ttest_expect_success POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n+\ttest_expect_success REFFILES,POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n \t\tmkdir sub && (\n \t\t\tcd sub &&\n \t\t\tumask $u &&\n@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' '\n+# For reftable, the check on .git/reftable/ is sufficient.\n+test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' '\n \tumask 077 &&\n \tgit config core.sharedRepository group &&\n \tgit reflog expire --all &&\n@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)'\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'template can set core.sharedrepository' '\n+test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' '\n \trm -rf child.git &&\n \tumask 0022 &&\n \tgit config core.sharedrepository 0666 &&\n-- \ngitgitgadget\n\n"},{"id":"432864","messageId":"c248b69c6e3ac90008b000f915b93877f349d769.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 24/25] t1401,t2011: parameterize HEAD.lock for REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:15Z","receivedAt":"2021-08-16T20:18:13Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1401-symbolic-ref.sh | 11 +++++++++--\n t/t2011-checkout-invalid-head.sh | 11 +++++++++--\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh\nindex 132a1b885ac..1b51013aded 100755\n--- a/t/t1401-symbolic-ref.sh\n+++ b/t/t1401-symbolic-ref.sh\n@@ -102,9 +102,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' '\n \ttest_cmp expect actual\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'symbolic-ref reports failure in exit code' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git symbolic-ref HEAD refs/heads/whatever\n '\n \ndiff --git a/t/t2011-checkout-invalid-head.sh b/t/t2011-checkout-invalid-head.sh\nindex e52022e1522..a56f7af442c 100755\n--- a/t/t2011-checkout-invalid-head.sh\n+++ b/t/t2011-checkout-invalid-head.sh\n@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' '\n \tgit checkout main --\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'checkout notices failure to lock HEAD' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git checkout -b other\n '\n \n-- \ngitgitgadget\n\n"},{"id":"432863","messageId":"9f1b792459e985dd7db74fa06229ef173993eafe.1629145037.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v2 25/25] t1404: annotate test cases with REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-16T20:17:16Z","receivedAt":"2021-08-16T20:18:14Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\n* Reftable for now lacks detailed error messages for directory/file conflicts.\n Skip message comparisons.\n\n* Mark tests that muck with .git directly as REFFILES.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1404-update-ref-errors.sh | 56 +++++++++++++++++++++++++++---------\n 1 file changed, 42 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t1404-update-ref-errors.sh b/t/t1404-update-ref-errors.sh\nindex b729c1f4803..811d5bb56d4 100755\n--- a/t/t1404-update-ref-errors.sh\n+++ b/t/t1404-update-ref-errors.sh\n@@ -27,7 +27,9 @@ test_update_rejected () {\n \tfi &&\n \tprintf \"create $prefix/%s $C\\n\" $create >input &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_i18ngrep -F \"$error\" output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_i18ngrep -F \"$error\" output.err\n+\tfi &&\n \tgit for-each-ref $prefix >actual &&\n \ttest_cmp unchanged actual\n }\n@@ -101,7 +103,9 @@ df_test() {\n \t\tprintf \"%s\\n\" \"delete $delname\" \"create $addname $D\"\n \tfi >commands &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_cmp expected-err output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected-err output.err\n+\tfi &&\n \tprintf \"%s\\n\" \"$C $delref\" >expected-refs &&\n \tgit for-each-ref --format=\"%(objectname) %(refname)\" $prefix/r >actual-refs &&\n \ttest_cmp expected-refs actual-refs\n@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks update' '\n@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks create' '\n@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/foo $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks delete' '\n@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/foo $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect update' '\n@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect update' '\n@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect create' '\n@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect delete' '\n@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect no-deref update' '\n@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref update' '\n@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect no-deref create' '\n@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n@@ -463,7 +489,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success REFFILES 'non-empty directory blocks create' '\n-- \ngitgitgadget\n"},{"id":"432866","messageId":"xmqqr1etqfnp.fsf@gitster.g","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/25] Support reftable ref backend for Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-16T20:48:58Z","receivedAt":"2021-08-16T20:49:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> This continues the work in https://github.com/git/git/pull/847, which the\n> gitgitgadget erroneously closed.\n\n> Changes relative to last series (version 20 Jul 2021, tip: 4a5891f )\n>\n> * fix HTTP serving\n> * fix t1501 (use absolute paths.)\n> * fix git-new-workdir\n> * fixes for FreeBSD (thanks, carenas@gmail.com!)\n\nThanks, will take a look later after the release is done; nice to\nsee these fix-up changes already applied in the series.\n"},{"id":"432881","messageId":"YRresPEnuv9EwBgS@carlos-mbp.lan","threadId":"56133","inReplyTo":"282d8e4d817f969a3c4d882cc833dd76f4451764.1629145036.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/25] reftable: (de)serialization for the polymorphic record type.","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-16T21:54:56Z","receivedAt":"2021-08-16T21:55:01Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Aug 16, 2021 at 08:16:58PM +0000, Han-Wen Nienhuys via GitGitGadget wrote:\n> From: Han-Wen Nienhuys \n...\n> +\tunion {\n> +\t\tstruct {\n> +\t\t\tuint8_t *new_hash;\n> +\t\t\tuint8_t *old_hash;\n> +\t\t\tchar *name;\n> +\t\t\tchar *email;\n> +\t\t\tuint64_t time;\n> +\t\t\tint16_t tz_offset;\n> +\t\t\tchar *message;\n> +\t\t} update;\n> +\t};\n\nthe use of an anonymous union here (which requires C11) could be made\nC89 compatible without any impact AFAIK by doing instead:\n\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nindex 7985b94ae2..136ff24910 100644\n--- a/reftable/reftable-record.h\n+++ b/reftable/reftable-record.h\n@@ -84,17 +84,15 @@ struct reftable_log_record {\n #define REFTABLE_NR_LOG_VALUETYPES 2\n \t} value_type;\n \n-\tunion {\n-\t\tstruct {\n-\t\t\tuint8_t *new_hash;\n-\t\t\tuint8_t *old_hash;\n-\t\t\tchar *name;\n-\t\t\tchar *email;\n-\t\t\tuint64_t time;\n-\t\t\tint16_t tz_offset;\n-\t\t\tchar *message;\n-\t\t} update;\n-\t};\n+\tstruct {\n+\t\tuint8_t *new_hash;\n+\t\tuint8_t *old_hash;\n+\t\tchar *name;\n+\t\tchar *email;\n+\t\tuint64_t time;\n+\t\tint16_t tz_offset;\n+\t\tchar *message;\n+\t} update;\n };\n \n /* returns whether 'ref' represents the deletion of a log record. */\n"},{"id":"432957","messageId":"cad3c4835a2945cfecc66da161fefa53d9c19574.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 01/25] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:43Z","receivedAt":"2021-08-17T13:41:50Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will simplify referencing them from code that is not deeply integrated with\nGit, in particular, the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n hash.h | 6 ++++++\n object-file.c | 7 ++-----\n 2 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9e25c40e9ac..5d40368f18a 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -95,12 +95,18 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n /* Number of algorithms supported (including unknown). */\n #define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n+/* \"sha1\", big-endian */\n+#define GIT_SHA1_FORMAT_ID 0x73686131\n+\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* \"s256\", big-endian */\n+#define GIT_SHA256_FORMAT_ID 0x73323536\n+\n /* The length in bytes and in hex digits of an object name (SHA-256 value). */\n #define GIT_SHA256_RAWSZ 32\n #define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\ndiff --git a/object-file.c b/object-file.c\nindex a8be8994814..7bfd5e6e2e9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -164,7 +164,6 @@ static void git_hash_unknown_final_oid(struct object_id *oid, git_hash_ctx *ctx)\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n-\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\tNULL,\n@@ -183,8 +182,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha1\",\n-\t\t/* \"sha1\", big-endian */\n-\t\t0x73686131,\n+\t\tGIT_SHA1_FORMAT_ID,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n \t\tGIT_SHA1_BLKSZ,\n@@ -199,8 +197,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha256\",\n-\t\t/* \"s256\", big-endian */\n-\t\t0x73323536,\n+\t\tGIT_SHA256_FORMAT_ID,\n \t\tGIT_SHA256_RAWSZ,\n \t\tGIT_SHA256_HEXSZ,\n \t\tGIT_SHA256_BLKSZ,\n-- \ngitgitgadget\n\n"},{"id":"432958","messageId":"79d4a23764de8e9aa7c0566ac4b151b4eaae7d73.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 06/25] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:48Z","receivedAt":"2021-08-17T13:41:51Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex addf2813086..86ba58cf140 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2447,6 +2447,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 00000000000..0044eecd9aa\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(!bs->ops);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(!bs->ops);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 00000000000..072e2727ad2\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 00000000000..5aa3990a573\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432961","messageId":"d92338467d66fcfedd57f209c97a798e9920d1e5.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 08/25] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:50Z","receivedAt":"2021-08-17T13:41:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be needed for reading reflog blocks in reftable.\n\nHelped-by: Carlo Marcelo Arenas Belón \nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 7 +++\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n config.mak.uname | 1 +\n configure.ac | 13 ++++++\n 6 files changed, 115 insertions(+)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n\ndiff --git a/Makefile b/Makefile\nindex e98d8ed17cf..16c883978d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -256,6 +256,8 @@ all::\n #\n # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n #\n+# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n+#\n # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n #\n@@ -1738,6 +1740,11 @@ ifdef NO_DEFLATE_BOUND\n \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n endif\n \n+ifdef NO_UNCOMPRESS2\n+\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n+\tREFTABLE_OBJS += compat/zlib-uncompress2.o\n+endif\n+\n ifdef NO_POSIX_GOODIES\n \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n endif\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..5711c63979d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -224,6 +224,7 @@ linux-gcc-default)\n \t;;\n Linux32)\n \tCC=gcc\n+\tMAKEFLAGS=\"$MAKEFLAGS NO_UNCOMPRESS2=1\"\n \t;;\n linux-musl)\n \tCC=gcc\ndiff --git a/compat/.gitattributes b/compat/.gitattributes\nnew file mode 100644\nindex 00000000000..40dbfb170da\n--- /dev/null\n+++ b/compat/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\nnew file mode 100644\nindex 00000000000..6893bb469ce\n--- /dev/null\n+++ b/compat/zlib-uncompress2.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress2 (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 69413fb3dc0..61e11550b1f 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -256,6 +256,7 @@ ifeq ($(uname_S),FreeBSD)\n \tFILENO_IS_A_MACRO = UnfortunatelyYes\n endif\n ifeq ($(uname_S),OpenBSD)\n+\tNO_UNCOMPRESS2 = YesPlease\n \tNO_STRCASESTR = YesPlease\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\ndiff --git a/configure.ac b/configure.ac\nindex 031e8d3fee8..c3a913103d0 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n LIBS=\"$old_LIBS\"\n \n+AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n+AC_LANG_PROGRAM([#include ],\n+ [uncompress2(NULL,NULL,NULL,NULL);])])\n+AC_MSG_CHECKING([for uncompress2 in -lz])\n+old_LIBS=\"$LIBS\"\n+LIBS=\"$LIBS -lz\"\n+AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n+\t[AC_MSG_RESULT([yes])],\n+\t[AC_MSG_RESULT([no])\n+\tNO_UNCOMPRESS2=yes])\n+LIBS=\"$old_LIBS\"\n+\n GIT_UNSTASH_FLAGS($ZLIB_PATH)\n \n GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n+GIT_CONF_SUBST([NO_UNCOMPRESS2])\n \n #\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n-- \ngitgitgadget\n\n"},{"id":"432959","messageId":"74ac0b135fac0ac61065cebe456de7b8d4e0eef8.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 09/25] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:51Z","receivedAt":"2021-08-17T13:41:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/block.c | 448 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 120 +++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 698 insertions(+)\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 16c883978d4..1d396c87be9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,10 +2454,12 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 00000000000..eb5268dd3a6\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,448 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \n+\n+#ifdef NO_UNCOMPRESS2\n+/*\n+ * This is uncompress2, which is only available in zlib >= 1.2.9\n+ * (released as of early 2017)\n+ */\n+int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n+\t\tuLong *sourceLen);\n+#endif\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK !=\n+\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n+\t\t\t\tblock->data + block_header_skip, &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp && source.ops)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 00000000000..e207706a644\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 00000000000..4b3ea262dcb\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,120 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b..c9deeaf08c7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432960","messageId":"2f2beb700781205bf750a53bbf6c04f42b9f0bd2.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 05/25] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:47Z","receivedAt":"2021-08-17T13:41:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 24 ++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 545 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d7..addf2813086 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -743,6 +743,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -821,6 +822,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1195,7 +1198,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2442,7 +2445,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2458,6 +2469,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2608,6 +2621,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2895,7 +2914,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3225,7 +3244,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 171b4124afe..c2bf5bdffc6 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -640,6 +640,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -662,7 +668,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -902,11 +908,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 00000000000..f761e48028c\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (!a) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 00000000000..096b36862b9\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 00000000000..1fcd2297256\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(!out[2]);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!out[1]);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 00000000000..bd0a02d3f68\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 00000000000..5f2185f1f34\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 00000000000..5e7698ae654\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 00000000000..4f62827b83b\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,24 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+/* This header glues the reftable library to the rest of Git */\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hash.h\" /* hash ID, sizes.*/\n+#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+\n+#include \n+\n+struct strbuf;\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 00000000000..84ac972cad0\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_FORMAT_ID));\n+}\n+\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add(b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 00000000000..774cb275bf6\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 00000000000..3b58e423e7b\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53a..f7c888ffda7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -53,13 +53,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f5228508..25f77469146 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -49,6 +49,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 00000000000..0ed14971a58\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"432962","messageId":"fca162dd67644bdc0945d015e5018f9371f40a7d.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 03/25] reftable: RFC: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:45Z","receivedAt":"2021-08-17T13:41:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe objective of this code is to be usable as a C library, so it can be reused\nin libgit2.\n\nThis is currently using a BSD license as it is the liberal license I could find,\nbut this could be changed to whatever fits the stated goal above.\n\nThis code is currently imported from github.com/hanwen/reftable. Once this code\nlands in git.git, the C code will be removed from github.com/hanwen/reftable,\nand the git.git code will be the source of truth.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 00000000000..402e0f9356b\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"432963","messageId":"44923074a5a926ba36f7321757738debdf8da412.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 04/25] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:46Z","receivedAt":"2021-08-17T13:42:00Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 00000000000..f6f16def921\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 00000000000..6f89bedf1a5\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432964","messageId":"e0f3476555ecf16ae31ee1bf87f00002dc8038af.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 12/25] reftable: generic interface to tables","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:54Z","receivedAt":"2021-08-17T13:42:02Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/generic.c | 169 +++++++++++++++++++++++++++++++++++\n reftable/generic.h | 32 +++++++\n reftable/reftable-generic.h | 47 ++++++++++\n reftable/reftable-iterator.h | 39 ++++++++\n reftable/reftable.c | 115 ++++++++++++++++++++++++\n 6 files changed, 405 insertions(+)\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex 9460f7de974..f45a981bc20 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,9 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n+REFTABLE_OBJS += reftable/generic.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/generic.c b/reftable/generic.c\nnew file mode 100644\nindex 00000000000..7a8a738d860\n--- /dev/null\n+++ b/reftable/generic.c\n@@ -0,0 +1,169 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = ~((uint64_t)0),\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_print(struct reftable_table *tab) {\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tuint32_t hash_id = reftable_table_hash_id(tab);\n+\tint err = reftable_table_seek_ref(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_table_seek_log(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn 0;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 00000000000..98886a06402\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 00000000000..d239751a778\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,47 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* dump table contents onto stdout for debugging */\n+int reftable_table_print(struct reftable_table *tab);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 00000000000..d3eee7af357\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 00000000000..0e4607a7cd6\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \ngitgitgadget\n\n"},{"id":"432965","messageId":"2c143c75cd54d8940f14026b7c6b481283a437e6.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 07/25] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:49Z","receivedAt":"2021-08-17T13:42:02Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1212 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 412 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1901 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 86ba58cf140..e98d8ed17cf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2449,7 +2449,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 00000000000..5eee72c4c11\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 00000000000..6a5dac32dc6\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = rec;\n+\tconst struct reftable_ref_record *src = src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release(rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tobj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));\n+\tCOPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->value.update.name,\n+\t\t log->value.update.email, log->value.update.time,\n+\t\t log->value.update.tz_offset);\n+\t\thex_format(hex, log->value.update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->value.update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->value.update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->value.update.email) {\n+\t\t\tdst->value.update.email =\n+\t\t\t\txstrdup(dst->value.update.email);\n+\t\t}\n+\t\tif (dst->value.update.name) {\n+\t\t\tdst->value.update.name =\n+\t\t\t\txstrdup(dst->value.update.name);\n+\t\t}\n+\t\tif (dst->value.update.message) {\n+\t\t\tdst->value.update.message =\n+\t\t\t\txstrdup(dst->value.update.message);\n+\t\t}\n+\n+\t\tif (dst->value.update.new_hash) {\n+\t\t\tdst->value.update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->value.update.new_hash,\n+\t\t\t src->value.update.new_hash, hash_size);\n+\t\t}\n+\t\tif (dst->value.update.old_hash) {\n+\t\t\tdst->value.update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->value.update.old_hash,\n+\t\t\t src->value.update.old_hash, hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->value.update.new_hash);\n+\t\treftable_free(r->value.update.old_hash);\n+\t\treftable_free(r->value.update.name);\n+\t\treftable_free(r->value.update.email);\n+\t\treftable_free(r->value.update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_log_record *r = rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->value.update.old_hash;\n+\tnewh = r->value.update.new_hash;\n+\tif (!oldh) {\n+\t\toldh = zero;\n+\t}\n+\tif (!newh) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->value.update.name ? r->value.update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->value.update.email ? r->value.update.email : \"\",\n+\t\t\t s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->value.update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->value.update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(\n+\t\tr->value.update.message ? r->value.update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->value.update.old_hash);\n+\t\t\tFREE_AND_NULL(r->value.update.new_hash);\n+\t\t\tFREE_AND_NULL(r->value.update.message);\n+\t\t\tFREE_AND_NULL(r->value.update.email);\n+\t\t\tFREE_AND_NULL(r->value.update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->value.update.old_hash =\n+\t\treftable_realloc(r->value.update.old_hash, hash_size);\n+\tr->value.update.new_hash =\n+\t\treftable_realloc(r->value.update.new_hash, hash_size);\n+\n+\tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.name =\n+\t\treftable_realloc(r->value.update.name, dest.len + 1);\n+\tmemcpy(r->value.update.name, dest.buf, dest.len);\n+\tr->value.update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.email =\n+\t\treftable_realloc(r->value.update.email, dest.len + 1);\n+\tmemcpy(r->value.update.email, dest.buf, dest.len);\n+\tr->value.update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->value.update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->value.update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.message =\n+\t\treftable_realloc(r->value.update.message, dest.len + 1);\n+\tmemcpy(r->value.update.message, dest.buf, dest.len);\n+\tr->value.update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (!a)\n+\t\ta = empty;\n+\n+\tif (!b)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (!a)\n+\t\ta = zero;\n+\n+\tif (!b)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->value.update.name, b->value.update.name) &&\n+\t\t a->value.update.time == b->value.update.time &&\n+\t\t a->value.update.tz_offset == b->value.update.tz_offset &&\n+\t\t null_streq(a->value.update.email,\n+\t\t\t\t b->value.update.email) &&\n+\t\t null_streq(a->value.update.message,\n+\t\t\t\t b->value.update.message) &&\n+\t\t zero_hash_eq(a->value.update.old_hash,\n+\t\t\t\t b->value.update.old_hash, hash_size) &&\n+\t\t zero_hash_eq(a->value.update.new_hash,\n+\t\t\t\t b->value.update.new_hash, hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_index_record *rec = r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = rec;\n+\tconst struct reftable_index_record *src = src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a && b)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tconst struct reftable_log_record *la = a;\n+\tconst struct reftable_log_record *lb = b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 00000000000..498e8c50bf4\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 00000000000..f4ad7cace41\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,412 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tint i;\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = {\n+\t\t\t\t.update = {\n+\t\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t\t.time = 1577123507,\n+\t\t\t\t\t.tz_offset = 100,\n+\t\t\t\t},\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].value.update.new_hash, 1);\n+\tset_test_hash(in[0].value.update.old_hash, 2);\n+\tfor (i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = {\n+\t\t\t\t.update = {\n+\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t\t},\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 00000000000..5370d2288c7\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t} value;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b..09d4b83ef9b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432966","messageId":"ac3063fd08c1958aff4921c02f824743bfc35948.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 11/25] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:53Z","receivedAt":"2021-08-17T13:42:04Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 148 ++++++++\n reftable/writer.c | 690 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex be14b66f9e3..9460f7de974 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2459,6 +2459,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 00000000000..af36462ced5\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \"reftable-record.h\"\n+\n+#include \n+#include /* ssize_t */\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 00000000000..3ca721e9f64\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,690 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\t\t\t 1 :\n+\t\t\t 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy(dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = -1;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->value.update.message;\n+\tif (!w->opts.exact_log_message && log->value.update.message) {\n+\t\tstrbuf_addstr(&cleaned_message, log->value.update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t/* multiple lines not allowed. */\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->value.update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->value.update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tif (arg->last) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 00000000000..09b88673d97\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tssize_t (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432967","messageId":"a221967007980ddb1499b006ff845f6862aefa42.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 10/25] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:52Z","receivedAt":"2021-08-17T13:42:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 1d396c87be9..be14b66f9e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,11 +2458,13 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 00000000000..82db7995dd6\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 00000000000..fbdd002e23a\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 00000000000..09a970e17b9\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = arg;\n+\tif (c->last) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7..050551fa698 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432968","messageId":"460658a13cb0fbba3921d18ba3c84083aba65d52.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 21/25] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:03Z","receivedAt":"2021-08-17T13:42:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex db7c0068fb5..4177274bea4 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"432969","messageId":"3d57f7c443082fd2a7f01aee003a9cd3ca2dd910.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 20/25] refs: RFC: Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:02Z","receivedAt":"2021-08-17T13:42:15Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see Documentation/technical/reftable.txt.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by setting GIT_TEST_REFTABLE in the environment. When\nGIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set.\n\nThere is no option to git-init for now, as the test suite still shows failures\nwith GIT_TEST_REFTABLE=1.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nHelped-by: Patrick Steinhardt \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 39 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1691 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/t0031-reftable.sh | 310 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 23 files changed, 2159 insertions(+), 29 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdca..82c5940f143 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffd..72576235833 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex 152bbd6c134..6d0074939e5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -990,6 +990,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66fe66679c8..baa1ff4fc60 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c2f03f6018e..22b07d2b2fb 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -226,6 +228,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -235,6 +238,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -249,9 +270,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -268,7 +286,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -383,7 +401,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -422,6 +440,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -476,6 +495,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -549,6 +571,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -713,5 +736,5 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..d43d6898039 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n static int do_clear_stash(void)\n {\n \tstruct object_id obj;\n+\tint result;\n \tif (get_oid(ref_stash, &obj))\n \t\treturn 0;\n \n-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n+\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n+\n+\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n+\t * even when refs are deleted. */\n+\tdelete_reflog(ref_stash);\n+\treturn result;\n }\n \n static int clear_stash(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 0d0a80da61f..6f4facebc61 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -328,9 +329,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex bd4869beee4..777d99013dd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -656,9 +656,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1076,6 +1077,7 @@ struct repository_format {\n \tint hash_algo;\n \tint sparse_index;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 61e11550b1f..019c88d5df4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -712,7 +712,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba1..1a25789d285 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ EOM\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ EOM\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a5215..989197aace0 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/refs.c b/refs.c\nindex 8b9f7c3a80a..00c4d24df6e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,15 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1880,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1902,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1962,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1976,7 @@ done:\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1990,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0..5a6d4ca9fa8 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 3155708345f..e36f215067e 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -672,6 +672,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..d972910bf21\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1691 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * two separate databases (ie. two reftable/ directories), one for the\n+ * repository, and one for the worktree. For reading, we merge the view (see\n+ * git_reftable_iterator) of both, when necessary.\n+ *\n+ * Unfortunately, the worktrees can also be selected by specifying a magic\n+ * refname (eg. worktree/BLA/refname, even if BLA isn't the current worktree.)\n+ */\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tconst char *wtname = refname;\n+\tint wtname_len = 0;\n+\tconst char *wtref = refname;\n+\n+\tif (refname == NULL)\n+\t\treturn store->main_stack;\n+\n+\tif (!parse_worktree_ref(refname, &wtname, &wtname_len, &wtref) &&\n+\t wtname_len) {\n+\t\t/* this makes me cry. Woe you if you try to access\n+\t\t * worktree/BLA/REF and the current worktree\n+\t\t * from the same process.\n+\t\t */\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_write_options cfg = {\n+\t\t\t.block_size = 4096,\n+\t\t\t.hash_id = the_hash_algo->format_id,\n+\t\t};\n+\n+\t\tstrbuf_addstr(&wt_dir, store->base.gitdir);\n+\t\tstrbuf_addstr(&wt_dir, \"/worktrees/\");\n+\t\tstrbuf_add(&wt_dir, wtname, wtname_len);\n+\t\tstrbuf_addstr(&wt_dir, \"/reftable\");\n+\n+\t\tif (store->worktree_stack)\n+\t\t\treftable_stack_destroy(store->worktree_stack);\n+\t\tstore->err = reftable_new_stack(&store->worktree_stack,\n+\t\t\t\t\t\twt_dir.buf, cfg);\n+\t\tassert(store->err != REFTABLE_API_ERROR);\n+\n+\t\treturn store->worktree_stack;\n+\t}\n+\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *out = ref;\n+\tint name_len = 0;\n+\tif (skip_prefix(ref, \"main-worktree/\", &out))\n+\t\treturn out;\n+\n+\tif (!parse_worktree_ref(ref, NULL, &name_len, &out) && name_len) {\n+\t\treturn out;\n+\t}\n+\n+\treturn ref;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tlog->value.update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path. Mutates its arg, trimming it to the\n+ * gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf gitdir = STRBUF_INIT;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_realpath(&wt_buf, path, /*die_on_error=*/0);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tstrbuf_addbuf(&gitdir, &wt_buf);\n+\t} else {\n+\t\tstrbuf_realpath(&gitdir, path, /*die_on_error=*/0);\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir.buf);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir.buf);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\n+\t\trefs->err =\n+\t\t\treftable_new_stack(&refs->worktree_stack, sb.buf, cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\tstrbuf_release(&gitdir);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs,\n+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_id = *null_oid();\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\tlog->value.update.message = u->msg;\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\n+\t\tlog->value.update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.value.update.message = xstrdup(arg->logmsg);\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.value.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.value.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.value.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.value.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.value.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.value.update.old_hash != NULL ||\n+\t\t log.value.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.value.update.message = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\told_ref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].value.update.message = (char *)arg->logmsg;\n+\t\ttodo[0].value.update.old_hash = val1;\n+\t\ttodo[0].value.update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].value.update.old_hash = NULL;\n+\t\ttodo[1].value.update.new_hash = val1;\n+\t\ttodo[1].value.update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* this copies the entire reflog history. Is this the right semantics?\n+\t */\n+\t/* XXX should clear out existing reflog entries for oldname? */\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\toidread(&ri->oid, ri->log.value.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.value.update.name,\n+\t\t\t\t\t log.value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log.value.update.time, log.value.update.tz_offset,\n+\t\t\t log.value.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->value.update.old_hash);\n+\t\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->value.update.name,\n+\t\t\t\t\t log->value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t\t log->value.update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].value.update.old_hash);\n+\t\toidread(&noid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash != NULL) {\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* XXX - skip writing records that were not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t} else {\n+\t\t/* XXX - print something */\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex b2bf44c6faf..f1cc8df47c4 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -180,6 +180,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex 3740c93bc0f..1bd9b4d09c8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -82,6 +82,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex eb9367ca5cb..58504dfeb68 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -498,6 +498,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -648,6 +651,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1312,6 +1316,8 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\tthe_repository->repository_format_partial_clone =\n \t\t\t\trepo_fmt.partial_clone;\n \t\t\trepo_fmt.partial_clone = NULL;\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n \t\t}\n \t}\n \t/*\n@@ -1399,6 +1405,8 @@ void check_repository_format(struct repository_format *fmt)\n \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tthe_repository->repository_format_partial_clone =\n \t\txstrdup_or_null(fmt->partial_clone);\n+\t/* XXX why is repo->ref_storage_format set in multiple places?! */\n+\tthe_repository->ref_storage_format = xstrdup_or_null(fmt->ref_storage);\n \tclear_repository_format(&repo_fmt);\n }\n \ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 00000000000..d2b398ba5dc\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,310 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+write_script fake_editor <<\\EOF\n+echo \"$MSG\" >\"$1\"\n+echo \"$MSG\" >&2\n+EOF\n+GIT_EDITOR=./fake_editor\n+export GIT_EDITOR\n+\n+\n+test_expect_success 'using reftable' '\n+\tinitialize &&\n+\ttest -d .git/reftable &&\n+\ttest -f .git/reftable/tables.list\n+'\n+\n+test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n+\tinitialize &&\n+\ttest_commit 1st &&\n+\ttest_commit 2nd &&\n+\tMSG=b4 git notes add &&\n+\tMSG=b3 git notes edit &&\n+\techo b4 >expect &&\n+\tgit notes --ref commits@{1} show >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git reflog delete' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_commit file2 &&\n+\ttest_commit file3 &&\n+\ttest_commit file4 &&\n+\tgit reflog delete HEAD@{1} &&\n+\tgit reflog > output &&\n+\t! grep file3 output\n+'\n+\n+test_expect_success 'branch -D delete nonexistent branch' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_must_fail git branch -D ../../my-private-file\n+'\n+\n+test_expect_success 'branch copy' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\ttest_commit file2 &&\n+\tgit branch src &&\n+\tgit reflog src > expect &&\n+\tgit branch -c src dst &&\n+\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'update-ref on corrupted data' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tOLD_SHA1=$(git rev-parse HEAD) &&\n+\ttest_commit file2 &&\n+\tls -l .git/reftable &&\n+\tfor f in .git/reftable/*.ref\n+\tdo\n+\t\t>$f\n+\tdone &&\n+\ttest_must_fail git update-ref refs/heads/main $OLD_SHA1\n+'\n+\n+test_expect_success 'git stash' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttouch actual expected &&\n+\tgit -c status.showStash=true status >expected &&\n+\techo hoi >> file.t &&\n+\tgit stash push -m stashed &&\n+\tgit stash clear &&\n+\tgit -c status.showStash=true status >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'rename branch' '\n+\tinitialize &&\n+\tgit symbolic-ref HEAD refs/heads/before &&\n+\ttest_commit file &&\n+\tgit show-ref | sed s/before/after/g > expected &&\n+\tgit branch -M after &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(GIT_TEST_REFTABLE=1 git_init || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\t(GIT_TEST_REFTABLE=1 git_init start) &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tGIT_TEST_REFTABLE=1 git init -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat << EOF >expect &&\n+HEAD\n+refs/heads/main\n+refs/tags/m1\n+EOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506..cdc21bf2dcb 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; need files backend'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..6bdd430dfe3 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 577f32dc71f..e523c3dd624 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; requires files ref backend'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..375e1dfd663 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1524,7 +1524,12 @@ parisc* | hppa*)\n \t;;\n esac\n \n-test_set_prereq REFFILES\n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq !REFFILES\n+else\n+ test_set_prereq REFFILES\n+fi\n \n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n-- \ngitgitgadget\n\n"},{"id":"432970","messageId":"88dbab683dc254f430ff70e852d65b1e41ef4959.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 19/25] reftable: add dump utility","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:01Z","receivedAt":"2021-08-17T13:42:20Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Carlo Marcelo Arenas Belón \n---\n reftable/dump.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 107 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 00000000000..155953d1b82\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,107 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"git-compat-util.h\"\n+#include \"hash.h\"\n+\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n+#include \"reftable-generic.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -6 sha256 hash format\\n\"\n+\t \" -h this help\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n+\t\t\tbreak;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n+\t\t\topt_dump_table = 1;\n+\t\telse if (!strcmp(\"-6\", argv[1]))\n+\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n+\t\t\topt_dump_stack = 1;\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n+\t\t\topt_compact = 1;\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t}\n+\n+\tif (argc != 2) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[1];\n+\n+\tif (opt_dump_table) {\n+\t\terr = reftable_reader_print_file(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = reftable_stack_print_directory(arg, opt_hash_id);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \ngitgitgadget\n\n"},{"id":"432971","messageId":"953590034a89a4ad22f02865981cbb6516107f4c.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 15/25] reftable: add a heap-based priority queue for reftable records","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:57Z","receivedAt":"2021-08-17T13:42:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5e2ff446894..744a63b9d2c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n@@ -2468,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 00000000000..8918d158e2d\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 00000000000..385d2fb139a\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 00000000000..ad21673e854\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 3d541fa5c0c..0019cbcfa49 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ https://developers.google.com/open-source/licenses/bsd\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int readwrite_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 898aba836fd..0b5a1701df1 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"432972","messageId":"f1fe338f67e875686333c7a3bfb6860ed7815f0b.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 13/25] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:55Z","receivedAt":"2021-08-17T13:42:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 194 +++++++++\n reftable/iter.h | 69 ++++\n reftable/reader.c | 801 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 +++\n reftable/reftable-reader.h | 101 +++++\n 6 files changed, 1233 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex f45a981bc20..5c857989091 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2456,7 +2456,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/generic.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 00000000000..93d04f735b8\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,194 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn !it->ops;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\treftable_free(it->offsets);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 00000000000..09eb0cbfa59\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,69 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+\n+/* Takes ownership of `offsets` */\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 00000000000..49f4ec070e6\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,801 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (!source->ops) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\tuint64_t file_size = block_source_size(source);\n+\n+\t/* Need +1 to read type of first block. */\n+\tuint32_t read_size = header_size(2) + 1; /* read v2 because it's larger. */\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\tif (read_size > file_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = block_source_read_block(source, &header, 0, read_size);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = file_size - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (!ti->bi.br) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next(ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek(tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id(tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+\n+int reftable_reader_print_file(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_reader *r = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\treftable_table_from_reader(&tab, r);\n+\terr = reftable_table_print(&tab);\n+done:\n+\treftable_reader_free(r);\n+\treturn err;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 00000000000..39583e5dbcd\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 00000000000..4a4bc2fdf85\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,101 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* print table onto stdout for debugging. */\n+int reftable_reader_print_file(const char *tablename);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"432973","messageId":"3110d6e6758a31aa3b72842dc664ca160f07814b.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 25/25] t1404: annotate test cases with REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:07Z","receivedAt":"2021-08-17T13:42:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\n* Reftable for now lacks detailed error messages for directory/file conflicts.\n Skip message comparisons.\n\n* Mark tests that muck with .git directly as REFFILES.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1404-update-ref-errors.sh | 56 +++++++++++++++++++++++++++---------\n 1 file changed, 42 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t1404-update-ref-errors.sh b/t/t1404-update-ref-errors.sh\nindex b729c1f4803..811d5bb56d4 100755\n--- a/t/t1404-update-ref-errors.sh\n+++ b/t/t1404-update-ref-errors.sh\n@@ -27,7 +27,9 @@ test_update_rejected () {\n \tfi &&\n \tprintf \"create $prefix/%s $C\\n\" $create >input &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_i18ngrep -F \"$error\" output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_i18ngrep -F \"$error\" output.err\n+\tfi &&\n \tgit for-each-ref $prefix >actual &&\n \ttest_cmp unchanged actual\n }\n@@ -101,7 +103,9 @@ df_test() {\n \t\tprintf \"%s\\n\" \"delete $delname\" \"create $addname $D\"\n \tfi >commands &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_cmp expected-err output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected-err output.err\n+\tfi &&\n \tprintf \"%s\\n\" \"$C $delref\" >expected-refs &&\n \tgit for-each-ref --format=\"%(objectname) %(refname)\" $prefix/r >actual-refs &&\n \ttest_cmp expected-refs actual-refs\n@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks update' '\n@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks create' '\n@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/foo $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks delete' '\n@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/foo $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect update' '\n@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect update' '\n@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect create' '\n@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect delete' '\n@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect no-deref update' '\n@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref update' '\n@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect no-deref create' '\n@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n@@ -463,7 +489,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success REFFILES 'non-empty directory blocks create' '\n-- \ngitgitgadget\n"},{"id":"432974","messageId":"64fe3c8edc5774847c33a612663e143a270b7b27.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 16/25] reftable: add merged table view","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:58Z","receivedAt":"2021-08-17T13:42:32Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 764 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 744a63b9d2c..defcc054fb6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2469,6 +2470,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 00000000000..e5b53da6db3\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,362 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (!mt) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record(tab, it, rec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 00000000000..8c4d4d58d77\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 00000000000..1e2afe37b8b\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,292 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n+\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == GIT_SHA1_FORMAT_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 00000000000..1a6d16915ab\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 0b5a1701df1..8087f2da4e6 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"432977","messageId":"764d45fccd4995ceefee26b2c340a54593f91994.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 18/25] reftable: implement stack, a mutable database of reftable files.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:00Z","receivedAt":"2021-08-17T13:42:33Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-stack.h | 128 ++++\n reftable/stack.c | 1396 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 948 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2515 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 04b143f670d..152bbd6c134 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2475,6 +2475,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 00000000000..1b602dda58a\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. After calling this,\n+ * reftable_addition_destroy should still be called.\n+ */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+/* print the entire stack represented by the directory */\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 00000000000..48e22a6c184\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1396 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t/* On Windows, can only unlink after closing. */\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!rd) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i]) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t/* On Windows, can only unlink after closing. */\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (!names[i]) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len]) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (!add) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->time > 0 &&\n+\t\t log.value.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (!expiry && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (!dir) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_merged_table *merged = NULL;\n+\tstruct reftable_table table = { NULL };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\treftable_table_from_merged_table(&table, merged);\n+\terr = reftable_table_print(&table);\n+done:\n+\tif (stack)\n+\t\treftable_stack_destroy(stack);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 00000000000..f57005846e5\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 00000000000..890a5c01990\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,948 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static char *get_tmp_dir(const char *prefix)\n+{\n+\tchar *dir = get_tmp_template(prefix);\n+\tEXPECT(mkdtemp(dir));\n+\treturn dir;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i]; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\tprintf(\"testing print functionality:\\n\");\n+\terr = reftable_stack_print_directory(dir, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_print_directory(dir, GIT_SHA256_FORMAT_ID);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\n+\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .value = { .update = {\n+\t\t\t\t\t\t\t\t.new_hash = h1,\n+\t\t\t\t\t\t\t\t.old_hash = h2,\n+\t\t\t\t\t\t\t} } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.value.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.value.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.value.update.message, \"one\\n\"));\n+\n+\tinput.value.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.value.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\t/* even entries add the refs, odd entries delete them. */\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].value.update.new_hash =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t\t\tlogs[i].value.update.email =\n+\t\t\t\txstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].value.update.time = i;\n+\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 100;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_reftable_stack_add);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c8db6852c35..996da85f7b5 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432975","messageId":"f0b7b1e2380b549ff3fdf65d0208723977d7597d.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 24/25] t1401,t2011: parameterize HEAD.lock for REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:06Z","receivedAt":"2021-08-17T13:42:34Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1401-symbolic-ref.sh | 11 +++++++++--\n t/t2011-checkout-invalid-head.sh | 11 +++++++++--\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh\nindex 132a1b885ac..1b51013aded 100755\n--- a/t/t1401-symbolic-ref.sh\n+++ b/t/t1401-symbolic-ref.sh\n@@ -102,9 +102,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' '\n \ttest_cmp expect actual\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'symbolic-ref reports failure in exit code' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git symbolic-ref HEAD refs/heads/whatever\n '\n \ndiff --git a/t/t2011-checkout-invalid-head.sh b/t/t2011-checkout-invalid-head.sh\nindex e52022e1522..a56f7af442c 100755\n--- a/t/t2011-checkout-invalid-head.sh\n+++ b/t/t2011-checkout-invalid-head.sh\n@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' '\n \tgit checkout main --\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'checkout notices failure to lock HEAD' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git checkout -b other\n '\n \n-- \ngitgitgadget\n\n"},{"id":"432976","messageId":"b305fcd49f1911edc723948d7b2a0a444572c054.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 22/25] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:04Z","receivedAt":"2021-08-17T13:42:35Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0031-reftable.sh | 6 ++++++\n 5 files changed, 14 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 6d0074939e5..98b8f1a1f9f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2471,6 +2471,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 996da85f7b5..26b03d7b789 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex f7c888ffda7..338a57b104d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -61,6 +61,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 25f77469146..48cee1f4a2d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nindex d2b398ba5dc..c76e9042e1d 100755\n--- a/t/t0031-reftable.sh\n+++ b/t/t0031-reftable.sh\n@@ -288,6 +288,12 @@ test_expect_success 'FETCH_HEAD' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'dump reftable' '\n+\tinitialize &&\n+\thash_id=$(git config extensions.objectformat) &&\n+\ttest-tool dump-reftable $(test \"${hash_id}\" = \"sha256\" && echo \"-6\") -s .git/reftable\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \ngitgitgadget\n\n"},{"id":"432978","messageId":"acb0d435a20b3106353abe20c16db9e96713e679.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 17/25] reftable: implement refname validation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:59Z","receivedAt":"2021-08-17T13:42:39Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 342 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex defcc054fb6..04b143f670d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2474,6 +2474,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 00000000000..95734969324\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 00000000000..a24b40fcb42\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 00000000000..8645cd93bbd\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 8087f2da4e6..c8db6852c35 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"432979","messageId":"a44956d7b08c6a7f73d2c6d2cb2e8bc3b4192c32.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 23/25] t1301: document what needs to be done for reftable","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:40:05Z","receivedAt":"2021-08-17T13:42:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1301-shared-repo.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh\nindex 84bf1970d8b..a5755b4a434 100755\n--- a/t/t1301-shared-repo.sh\n+++ b/t/t1301-shared-repo.sh\n@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' '\n \t)\n '\n \n+# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/\n for u in 002 022\n do\n-\ttest_expect_success POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n+\ttest_expect_success REFFILES,POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n \t\tmkdir sub && (\n \t\t\tcd sub &&\n \t\t\tumask $u &&\n@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' '\n+# For reftable, the check on .git/reftable/ is sufficient.\n+test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' '\n \tumask 077 &&\n \tgit config core.sharedRepository group &&\n \tgit reflog expire --all &&\n@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)'\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'template can set core.sharedrepository' '\n+test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' '\n \trm -rf child.git &&\n \tumask 0022 &&\n \tgit config core.sharedrepository 0666 &&\n-- \ngitgitgadget\n\n"},{"id":"432980","messageId":"7da9a85c1db8df3962c8e50dabef424dd82fb6c9.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 14/25] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:56Z","receivedAt":"2021-08-17T13:42:43Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/readwrite_test.c | 652 ++++++++++++++++++++++++++++++++++++++\n reftable/reftable-tests.h | 2 +-\n t/helper/test-reftable.c | 1 +\n 4 files changed, 655 insertions(+), 1 deletion(-)\n create mode 100644 reftable/readwrite_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5c857989091..5e2ff446894 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2469,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nnew file mode 100644\nindex 00000000000..5f6bcc2f775\n--- /dev/null\n+++ b/reftable/readwrite_test.c\n@@ -0,0 +1,652 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.value.update.new_hash = hash;\n+\t\tlog.value.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size(\n+\t\t\t\t(hash_id == GIT_SHA256_FORMAT_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record\n+\t\tlog = { .refname = \"refs/heads/master\",\n+\t\t\t.update_index = 0xa,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = { .update = {\n+\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t } } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.value.update.old_hash = hash1;\n+\tlog.value.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.value.update.old_hash = hash1;\n+\t\tlog.value.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n+\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_write_empty_table(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_write_key_order(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record refs[2] = {\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}, {\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}\n+\t};\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\terr = reftable_writer_add_ref(w, &refs[0]);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_add_ref(w, &refs[1]);\n+\tprintf(\"%d\\n\", err);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_writer_close(w);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_corrupt_table_empty(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+}\n+\n+static void test_corrupt_table(void)\n+{\n+\tuint8_t zeros[1024] = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\tstrbuf_add(&buf, zeros, sizeof(zeros));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\tstrbuf_release(&buf);\n+}\n+\n+int readwrite_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_corrupt_table);\n+\tRUN_TEST(test_corrupt_table_empty);\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_write_key_order);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_write_empty_table);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654..3d541fa5c0c 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n-int reftable_test_main(int argc, const char **argv);\n+int readwrite_test_main(int argc, const char **argv);\n int stack_test_main(int argc, const char **argv);\n int tree_test_main(int argc, const char **argv);\n int reftable_dump_main(int argc, char *const *argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa698..898aba836fd 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"432982","messageId":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v2.git.git.1629145036.gitgitgadget@gmail.com","subject":"[PATCH v3 00/25] Support reftable ref backend for Git","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:42Z","receivedAt":"2021-08-17T13:44:18Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This continues the work in https://github.com/git/git/pull/847, which the\ngitgitgadget erroneously closed.\n\nChanges relative to last series (version 20 Jul 2021, tip: 4a5891f )\n\n * fix HTTP serving\n * fix t1501 (use absolute paths.)\n * fix git-new-workdir\n * fixes for FreeBSD (thanks, carenas@gmail.com!)\n\nFor the 'seen' branch, the following should be applied:\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d7137d1213..9323931eeb 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type);\n+ unsigned int *type, int *failure_errno);\n \n static void clear_reftable_log_record(struct reftable_log_record *log)\n {\n@@ -425,13 +424,14 @@ static int fixup_symrefs(struct ref_store *ref_store,\n for (i = 0; i < transaction->nr; i++) {\n struct ref_update *update = transaction->updates[i];\n struct object_id old_oid;\n+ int failure_errno;\n \n err = git_reftable_read_raw_ref(ref_store, update->refname,\n &old_oid, &referent,\n /* mutate input, like\n files-backend.c */\n- &update->type);\n- if (err < 0 && errno == ENOENT &&\n+ &update->type, &failure_errno);\n+ if (err < 0 && failure_errno == ENOENT &&\n is_null_oid(&update->old_oid)) {\n err = 0;\n }\n@@ -1602,7 +1603,7 @@ static int reftable_error_to_errno(int err)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n const char *refname, struct object_id *oid,\n struct strbuf *referent,\n- unsigned int *type)\n+ unsigned int *type, int *failure_errno)\n {\n struct git_reftable_ref_store *refs =\n (struct git_reftable_ref_store *)ref_store;\n@@ -1626,13 +1627,11 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \n err = reftable_stack_read_ref(stack, refname, &ref);\n if (err > 0) {\n- errno = ENOENT;\n+ *failure_errno = ENOENT;\n err = -1;\n goto done;\n }\n if (err < 0) {\n- errno = reftable_error_to_errno(err);\n- err = -1;\n goto done;\n }\n\n\nHan-Wen Nienhuys (24):\n hash.h: provide constants for the hash IDs\n init-db: set the_repository->hash_algo early on\n reftable: RFC: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n Provide zlib's uncompress2 from compat/zlib-compat.c\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n refs: RFC: Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n t1301: document what needs to be done for reftable\n t1401,t2011: parameterize HEAD.lock for REFFILES\n t1404: annotate test cases with REFFILES\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 54 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 60 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +\n config.mak.uname | 3 +-\n configure.ac | 13 +\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n contrib/workdir/git-new-workdir | 2 +-\n hash.h | 6 +\n object-file.c | 7 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1691 +++++++++++++++++\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 +\n reftable/block.c | 448 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 120 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 107 ++\n reftable/error.c | 41 +\n reftable/generic.c | 169 ++\n reftable/generic.h | 32 +\n reftable/iter.c | 194 ++\n reftable/iter.h | 69 +\n reftable/merged.c | 362 ++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 292 +++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 801 ++++++++\n reftable/reader.h | 66 +\n reftable/readwrite_test.c | 652 +++++++\n reftable/record.c | 1212 ++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 412 ++++\n reftable/refname.c | 209 ++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 47 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 101 +\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 128 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 148 ++\n reftable/reftable.c | 115 ++\n reftable/stack.c | 1396 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 948 +++++++++\n reftable/system.h | 24 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 690 +++++++\n reftable/writer.h | 50 +\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 316 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1301-shared-repo.sh | 8 +-\n t/t1401-symbolic-ref.sh | 11 +-\n t/t1404-update-ref-errors.sh | 56 +-\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t2011-checkout-invalid-head.sh | 11 +-\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 94 files changed, 13178 insertions(+), 62 deletions(-)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/readwrite_test.c\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1054%2Fhanwen%2Flibreftable-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1054/hanwen/libreftable-v3\nPull-Request: https://github.com/git/git/pull/1054\n\nRange-diff vs v2:\n\n 1: c40393e83e1 = 1: cad3c4835a2 hash.h: provide constants for the hash IDs\n 2: 49dffeeae58 = 2: 97aba3ae406 init-db: set the_repository->hash_algo early on\n 3: dba354eca12 = 3: fca162dd676 reftable: RFC: add LICENSE\n 4: 75864034af2 = 4: 44923074a5a reftable: add error related functionality\n 5: eabb25d95b3 ! 5: 2f2beb70078 reftable: utility functions\n @@ reftable/system.h (new)\n +#ifndef SYSTEM_H\n +#define SYSTEM_H\n +\n -+// This header glues the reftable library to the rest of Git\n ++/* This header glues the reftable library to the rest of Git */\n +\n +#include \"git-compat-util.h\"\n +#include \"strbuf.h\"\n 6: ad54d17f84b = 6: 79d4a23764d reftable: add blocksource, an abstraction for random access reads\n 7: 282d8e4d817 ! 7: 2c143c75cd5 reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record.c (new)\n +\t\tbreak;\n +\tcase REFTABLE_LOG_UPDATE:\n +\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n -+\t\t log->refname, log->update_index, log->update.name,\n -+\t\t log->update.email, log->update.time,\n -+\t\t log->update.tz_offset);\n -+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n ++\t\t log->refname, log->update_index, log->value.update.name,\n ++\t\t log->value.update.email, log->value.update.time,\n ++\t\t log->value.update.tz_offset);\n ++\t\thex_format(hex, log->value.update.old_hash, hash_size(hash_id));\n +\t\tprintf(\"%s => \", hex);\n -+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n -+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n ++\t\thex_format(hex, log->value.update.new_hash, hash_size(hash_id));\n ++\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->value.update.message);\n +\t\tbreak;\n +\t}\n +}\n @@ reftable/record.c (new)\n +\tcase REFTABLE_LOG_DELETION:\n +\t\tbreak;\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\tif (dst->update.email) {\n -+\t\t\tdst->update.email = xstrdup(dst->update.email);\n ++\t\tif (dst->value.update.email) {\n ++\t\t\tdst->value.update.email =\n ++\t\t\t\txstrdup(dst->value.update.email);\n +\t\t}\n -+\t\tif (dst->update.name) {\n -+\t\t\tdst->update.name = xstrdup(dst->update.name);\n ++\t\tif (dst->value.update.name) {\n ++\t\t\tdst->value.update.name =\n ++\t\t\t\txstrdup(dst->value.update.name);\n +\t\t}\n -+\t\tif (dst->update.message) {\n -+\t\t\tdst->update.message = xstrdup(dst->update.message);\n ++\t\tif (dst->value.update.message) {\n ++\t\t\tdst->value.update.message =\n ++\t\t\t\txstrdup(dst->value.update.message);\n +\t\t}\n +\n -+\t\tif (dst->update.new_hash) {\n -+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n -+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n -+\t\t\t hash_size);\n ++\t\tif (dst->value.update.new_hash) {\n ++\t\t\tdst->value.update.new_hash = reftable_malloc(hash_size);\n ++\t\t\tmemcpy(dst->value.update.new_hash,\n ++\t\t\t src->value.update.new_hash, hash_size);\n +\t\t}\n -+\t\tif (dst->update.old_hash) {\n -+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n -+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n -+\t\t\t hash_size);\n ++\t\tif (dst->value.update.old_hash) {\n ++\t\t\tdst->value.update.old_hash = reftable_malloc(hash_size);\n ++\t\t\tmemcpy(dst->value.update.old_hash,\n ++\t\t\t src->value.update.old_hash, hash_size);\n +\t\t}\n +\t\tbreak;\n +\t}\n @@ reftable/record.c (new)\n +\tcase REFTABLE_LOG_DELETION:\n +\t\tbreak;\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\treftable_free(r->update.new_hash);\n -+\t\treftable_free(r->update.old_hash);\n -+\t\treftable_free(r->update.name);\n -+\t\treftable_free(r->update.email);\n -+\t\treftable_free(r->update.message);\n ++\t\treftable_free(r->value.update.new_hash);\n ++\t\treftable_free(r->value.update.old_hash);\n ++\t\treftable_free(r->value.update.name);\n ++\t\treftable_free(r->value.update.email);\n ++\t\treftable_free(r->value.update.message);\n +\t\tbreak;\n +\t}\n +\tmemset(r, 0, sizeof(struct reftable_log_record));\n @@ reftable/record.c (new)\n +\tif (reftable_log_record_is_deletion(r))\n +\t\treturn 0;\n +\n -+\toldh = r->update.old_hash;\n -+\tnewh = r->update.new_hash;\n ++\toldh = r->value.update.old_hash;\n ++\tnewh = r->value.update.new_hash;\n +\tif (!oldh) {\n +\t\toldh = zero;\n +\t}\n @@ reftable/record.c (new)\n +\tmemcpy(s.buf + hash_size, newh, hash_size);\n +\tstring_view_consume(&s, 2 * hash_size);\n +\n -+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n ++\tn = encode_string(r->value.update.name ? r->value.update.name : \"\", s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n +\n -+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n ++\tn = encode_string(r->value.update.email ? r->value.update.email : \"\",\n ++\t\t\t s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n +\n -+\tn = put_var_int(&s, r->update.time);\n ++\tn = put_var_int(&s, r->value.update.time);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n @@ reftable/record.c (new)\n +\tif (s.len < 2)\n +\t\treturn -1;\n +\n -+\tput_be16(s.buf, r->update.tz_offset);\n ++\tput_be16(s.buf, r->value.update.tz_offset);\n +\tstring_view_consume(&s, 2);\n +\n -+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n ++\tn = encode_string(\n ++\t\tr->value.update.message ? r->value.update.message : \"\", s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n @@ reftable/record.c (new)\n +\tif (val_type != r->value_type) {\n +\t\tswitch (r->value_type) {\n +\t\tcase REFTABLE_LOG_UPDATE:\n -+\t\t\tFREE_AND_NULL(r->update.old_hash);\n -+\t\t\tFREE_AND_NULL(r->update.new_hash);\n -+\t\t\tFREE_AND_NULL(r->update.message);\n -+\t\t\tFREE_AND_NULL(r->update.email);\n -+\t\t\tFREE_AND_NULL(r->update.name);\n ++\t\t\tFREE_AND_NULL(r->value.update.old_hash);\n ++\t\t\tFREE_AND_NULL(r->value.update.new_hash);\n ++\t\t\tFREE_AND_NULL(r->value.update.message);\n ++\t\t\tFREE_AND_NULL(r->value.update.email);\n ++\t\t\tFREE_AND_NULL(r->value.update.name);\n +\t\t\tbreak;\n +\t\tcase REFTABLE_LOG_DELETION:\n +\t\t\tbreak;\n @@ reftable/record.c (new)\n +\tif (in.len < 2 * hash_size)\n +\t\treturn REFTABLE_FORMAT_ERROR;\n +\n -+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n -+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n ++\tr->value.update.old_hash =\n ++\t\treftable_realloc(r->value.update.old_hash, hash_size);\n ++\tr->value.update.new_hash =\n ++\t\treftable_realloc(r->value.update.new_hash, hash_size);\n +\n -+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n -+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n ++\tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n ++\tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n +\n +\tstring_view_consume(&in, 2 * hash_size);\n +\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n -+\tmemcpy(r->update.name, dest.buf, dest.len);\n -+\tr->update.name[dest.len] = 0;\n ++\tr->value.update.name =\n ++\t\treftable_realloc(r->value.update.name, dest.len + 1);\n ++\tmemcpy(r->value.update.name, dest.buf, dest.len);\n ++\tr->value.update.name[dest.len] = 0;\n +\n +\tstrbuf_reset(&dest);\n +\tn = decode_string(&dest, in);\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n -+\tmemcpy(r->update.email, dest.buf, dest.len);\n -+\tr->update.email[dest.len] = 0;\n ++\tr->value.update.email =\n ++\t\treftable_realloc(r->value.update.email, dest.len + 1);\n ++\tmemcpy(r->value.update.email, dest.buf, dest.len);\n ++\tr->value.update.email[dest.len] = 0;\n +\n +\tts = 0;\n +\tn = get_var_int(&ts, &in);\n +\tif (n < 0)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n -+\tr->update.time = ts;\n ++\tr->value.update.time = ts;\n +\tif (in.len < 2)\n +\t\tgoto done;\n +\n -+\tr->update.tz_offset = get_be16(in.buf);\n ++\tr->value.update.tz_offset = get_be16(in.buf);\n +\tstring_view_consume(&in, 2);\n +\n +\tstrbuf_reset(&dest);\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n -+\tmemcpy(r->update.message, dest.buf, dest.len);\n -+\tr->update.message[dest.len] = 0;\n ++\tr->value.update.message =\n ++\t\treftable_realloc(r->value.update.message, dest.len + 1);\n ++\tmemcpy(r->value.update.message, dest.buf, dest.len);\n ++\tr->value.update.message[dest.len] = 0;\n +\n +\tstrbuf_release(&dest);\n +\treturn start.len - in.len;\n @@ reftable/record.c (new)\n +\tcase REFTABLE_LOG_DELETION:\n +\t\treturn 1;\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\treturn null_streq(a->update.name, b->update.name) &&\n -+\t\t a->update.time == b->update.time &&\n -+\t\t a->update.tz_offset == b->update.tz_offset &&\n -+\t\t null_streq(a->update.email, b->update.email) &&\n -+\t\t null_streq(a->update.message, b->update.message) &&\n -+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n -+\t\t\t\t hash_size) &&\n -+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n -+\t\t\t\t hash_size);\n ++\t\treturn null_streq(a->value.update.name, b->value.update.name) &&\n ++\t\t a->value.update.time == b->value.update.time &&\n ++\t\t a->value.update.tz_offset == b->value.update.tz_offset &&\n ++\t\t null_streq(a->value.update.email,\n ++\t\t\t\t b->value.update.email) &&\n ++\t\t null_streq(a->value.update.message,\n ++\t\t\t\t b->value.update.message) &&\n ++\t\t zero_hash_eq(a->value.update.old_hash,\n ++\t\t\t\t b->value.update.old_hash, hash_size) &&\n ++\t\t zero_hash_eq(a->value.update.new_hash,\n ++\t\t\t\t b->value.update.new_hash, hash_size);\n +\t}\n +\n +\tabort();\n @@ reftable/record_test.c (new)\n +\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n +\t\t\t.update_index = 42,\n +\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n -+\t\t\t.update = {\n -+\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n -+\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n -+\t\t\t\t.name = xstrdup(\"han-wen\"),\n -+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n -+\t\t\t\t.message = xstrdup(\"test\"),\n -+\t\t\t\t.time = 1577123507,\n -+\t\t\t\t.tz_offset = 100,\n ++\t\t\t.value = {\n ++\t\t\t\t.update = {\n ++\t\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t\t.name = xstrdup(\"han-wen\"),\n ++\t\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n ++\t\t\t\t\t.message = xstrdup(\"test\"),\n ++\t\t\t\t\t.time = 1577123507,\n ++\t\t\t\t\t.tz_offset = 100,\n ++\t\t\t\t},\n +\t\t\t}\n +\t\t},\n +\t\t{\n @@ reftable/record_test.c (new)\n +\t\t\t.value_type = REFTABLE_LOG_DELETION,\n +\t\t}\n +\t};\n -+\tset_test_hash(in[0].update.new_hash, 1);\n -+\tset_test_hash(in[0].update.old_hash, 2);\n ++\tset_test_hash(in[0].value.update.new_hash, 1);\n ++\tset_test_hash(in[0].value.update.old_hash, 2);\n +\tfor (i = 0; i < ARRAY_SIZE(in); i++) {\n +\t\tstruct reftable_record rec = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n @@ reftable/record_test.c (new)\n +\t\tstruct reftable_log_record out = {\n +\t\t\t.refname = xstrdup(\"old name\"),\n +\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n -+\t\t\t.update = {\n -+\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n -+\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n -+\t\t\t\t.name = xstrdup(\"old name\"),\n -+\t\t\t\t.email = xstrdup(\"old@email\"),\n -+\t\t\t\t.message = xstrdup(\"old message\"),\n ++\t\t\t.value = {\n ++\t\t\t\t.update = {\n ++\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t\t.name = xstrdup(\"old name\"),\n ++\t\t\t\t\t.email = xstrdup(\"old@email\"),\n ++\t\t\t\t\t.message = xstrdup(\"old message\"),\n ++\t\t\t\t},\n +\t\t\t},\n +\t\t};\n +\t\tstruct reftable_record rec_out = { NULL };\n @@ reftable/reftable-record.h (new)\n +\t\t\tint16_t tz_offset;\n +\t\t\tchar *message;\n +\t\t} update;\n -+\t};\n ++\t} value;\n +};\n +\n +/* returns whether 'ref' represents the deletion of a log record. */\n 8: 893ee4ca21b = 8: d92338467d6 Provide zlib's uncompress2 from compat/zlib-compat.c\n 9: e4da0a2ef37 ! 9: 74ac0b135fa reftable: reading/writing blocks\n @@ reftable/block_test.c (new)\n +#include \"block.h\"\n +\n +#include \"system.h\"\n -+\n +#include \"blocksource.h\"\n +#include \"basics.h\"\n +#include \"constants.h\"\n 10: 0f325881389 = 10: a2219670079 reftable: a generic binary tree implementation\n 11: 26cbc468f1f ! 11: ac3063fd08c reftable: write reftable files\n @@ reftable/writer.c (new)\n +\tif (log->refname == NULL)\n +\t\treturn REFTABLE_API_ERROR;\n +\n -+\tinput_log_message = log->update.message;\n -+\tif (!w->opts.exact_log_message && log->update.message) {\n -+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n ++\tinput_log_message = log->value.update.message;\n ++\tif (!w->opts.exact_log_message && log->value.update.message) {\n ++\t\tstrbuf_addstr(&cleaned_message, log->value.update.message);\n +\t\twhile (cleaned_message.len &&\n +\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n +\t\t\tstrbuf_setlen(&cleaned_message,\n +\t\t\t\t cleaned_message.len - 1);\n +\t\tif (strchr(cleaned_message.buf, '\\n')) {\n -+\t\t\t// multiple lines not allowed.\n ++\t\t\t/* multiple lines not allowed. */\n +\t\t\terr = REFTABLE_API_ERROR;\n +\t\t\tgoto done;\n +\t\t}\n +\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n -+\t\tlog->update.message = cleaned_message.buf;\n ++\t\tlog->value.update.message = cleaned_message.buf;\n +\t}\n +\n +\terr = reftable_writer_add_log_verbatim(w, log);\n -+\tlog->update.message = input_log_message;\n ++\tlog->value.update.message = input_log_message;\n +done:\n +\tstrbuf_release(&cleaned_message);\n +\treturn err;\n 12: 69129fca086 = 12: e0f3476555e reftable: generic interface to tables\n 13: 22599d785b7 = 13: f1fe338f67e reftable: read reftable files\n 14: 45565bfb9c4 ! 14: 7da9a85c1db reftable: reftable file level tests\n @@ reftable/readwrite_test.c (new)\n +\t\tlog.refname = name;\n +\t\tlog.update_index = update_index;\n +\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n -+\t\tlog.update.new_hash = hash;\n -+\t\tlog.update.message = \"message\";\n ++\t\tlog.value.update.new_hash = hash;\n ++\t\tlog.value.update.message = \"message\";\n +\n +\t\tn = reftable_writer_add_log(w, &log);\n +\t\tEXPECT(n == 0);\n @@ reftable/readwrite_test.c (new)\n +\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n +\t\tint off = i * opts.block_size;\n +\t\tif (off == 0) {\n -+\t\t\toff = header_size((hash_id == GIT_SHA256_FORMAT_ID) ? 2 :\n -+\t\t\t\t\t\t\t\t\t\t 1);\n ++\t\t\toff = header_size(\n ++\t\t\t\t(hash_id == GIT_SHA256_FORMAT_ID) ? 2 : 1);\n +\t\t}\n +\t\tEXPECT(buf->buf[off] == 'r');\n +\t}\n @@ reftable/readwrite_test.c (new)\n +\t};\n +\tint err;\n +\tint i;\n -+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n -+\t\t\t\t\t .update_index = 0xa,\n -+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n -+\t\t\t\t\t .update = {\n -+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n -+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n -+\t\t\t\t\t\t .tz_offset = 100,\n -+\t\t\t\t\t\t .time = 0x5e430672,\n -+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n -+\t\t\t\t\t } };\n ++\tstruct reftable_log_record\n ++\t\tlog = { .refname = \"refs/heads/master\",\n ++\t\t\t.update_index = 0xa,\n ++\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n ++\t\t\t.value = { .update = {\n ++\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n ++\t\t\t\t\t .email = \"hanwen@google.com\",\n ++\t\t\t\t\t .tz_offset = 100,\n ++\t\t\t\t\t .time = 0x5e430672,\n ++\t\t\t\t\t .message = \"commit: 9\\n\",\n ++\t\t\t\t } } };\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n +\n @@ reftable/readwrite_test.c (new)\n +\t\thash1[i] = (uint8_t)(rand() % 256);\n +\t\thash2[i] = (uint8_t)(rand() % 256);\n +\t}\n -+\tlog.update.old_hash = hash1;\n -+\tlog.update.new_hash = hash2;\n ++\tlog.value.update.old_hash = hash1;\n ++\tlog.value.update.new_hash = hash2;\n +\treftable_writer_set_limits(w, update_index, update_index);\n +\terr = reftable_writer_add_log(w, &log);\n +\tEXPECT_ERR(err);\n @@ reftable/readwrite_test.c (new)\n +\t\tlog.refname = names[i];\n +\t\tlog.update_index = i;\n +\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n -+\t\tlog.update.old_hash = hash1;\n -+\t\tlog.update.new_hash = hash2;\n ++\t\tlog.value.update.old_hash = hash1;\n ++\t\tlog.value.update.new_hash = hash2;\n +\n +\t\terr = reftable_writer_add_log(w, &log);\n +\t\tEXPECT_ERR(err);\n 15: 57b3e709c3e = 15: 953590034a8 reftable: add a heap-based priority queue for reftable records\n 16: f834cc5299f = 16: 64fe3c8edc5 reftable: add merged table view\n 17: 15799d93ff7 = 17: acb0d435a20 reftable: implement refname validation\n 18: b7728ceaef2 ! 18: 764d45fccd4 reftable: implement stack, a mutable database of reftable files.\n @@ reftable/stack.c (new)\n +\t\t\treftable_reader_free(st->readers[i]);\n +\n +\t\t\tif (filename.len) {\n -+\t\t\t\t// On Windows, can only unlink after closing.\n ++\t\t\t\t/* On Windows, can only unlink after closing. */\n +\t\t\t\tunlink(filename.buf);\n +\t\t\t}\n +\t\t}\n @@ reftable/stack.c (new)\n +\t\t\treader_close(cur[i]);\n +\t\t\treftable_reader_free(cur[i]);\n +\n -+\t\t\t// On Windows, can only unlink after closing.\n ++\t\t\t/* On Windows, can only unlink after closing. */\n +\t\t\tunlink(filename.buf);\n +\n +\t\t\tstrbuf_release(&filename);\n @@ reftable/stack.c (new)\n +\t\t}\n +\n +\t\tif (config && config->time > 0 &&\n -+\t\t log.update.time < config->time) {\n ++\t\t log.value.update.time < config->time) {\n +\t\t\tcontinue;\n +\t\t}\n +\n @@ reftable/stack_test.c (new)\n +\t\tlogs[i].update_index = N + i + 1;\n +\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n +\n -+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n -+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n -+\t\tset_test_hash(logs[i].update.new_hash, i);\n ++\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n ++\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n ++\t\tset_test_hash(logs[i].value.update.new_hash, i);\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_log_record input = { .refname = \"branch\",\n +\t\t\t\t\t .update_index = 1,\n +\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n -+\t\t\t\t\t .update = {\n -+\t\t\t\t\t\t .new_hash = h1,\n -+\t\t\t\t\t\t .old_hash = h2,\n -+\t\t\t\t\t } };\n ++\t\t\t\t\t .value = { .update = {\n ++\t\t\t\t\t\t\t\t.new_hash = h1,\n ++\t\t\t\t\t\t\t\t.old_hash = h2,\n ++\t\t\t\t\t\t\t} } };\n +\tstruct reftable_log_record dest = {\n +\t\t.update_index = 0,\n +\t};\n @@ reftable/stack_test.c (new)\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n -+\tinput.update.message = \"one\\ntwo\";\n ++\tinput.value.update.message = \"one\\ntwo\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT(err == REFTABLE_API_ERROR);\n +\n -+\tinput.update.message = \"one\";\n ++\tinput.value.update.message = \"one\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n ++\tEXPECT(0 == strcmp(dest.value.update.message, \"one\\n\"));\n +\n -+\tinput.update.message = \"two\\n\";\n ++\tinput.value.update.message = \"two\\n\";\n +\targ.update_index = 2;\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT_ERR(err);\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n ++\tEXPECT(0 == strcmp(dest.value.update.message, \"two\\n\"));\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n @@ reftable/stack_test.c (new)\n +\t\tlogs[i].update_index = 42;\n +\t\tif (i % 2 == 0) {\n +\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n -+\t\t\tlogs[i].update.new_hash =\n ++\t\t\tlogs[i].value.update.new_hash =\n +\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n -+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n -+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n ++\t\t\tset_test_hash(logs[i].value.update.new_hash, i);\n ++\t\t\tlogs[i].value.update.email =\n ++\t\t\t\txstrdup(\"identity@invalid\");\n +\t\t}\n +\t}\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\tlogs[i].update_index = i;\n +\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n -+\t\tlogs[i].update.time = i;\n -+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n -+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n -+\t\tset_test_hash(logs[i].update.new_hash, i);\n ++\t\tlogs[i].value.update.time = i;\n ++\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n ++\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n ++\t\tset_test_hash(logs[i].value.update.new_hash, i);\n +\t}\n +\n +\tfor (i = 1; i <= N; i++) {\n 19: 0f682be20bd = 19: 88dbab683dc reftable: add dump utility\n 20: 754085752f4 ! 20: 3d57f7c4430 refs: RFC: Reftable support for git-core\n @@ refs/reftable-backend.c (new)\n +\tlog->refname = NULL;\n +\tswitch (log->value_type) {\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\tlog->update.old_hash = NULL;\n -+\t\tlog->update.new_hash = NULL;\n -+\t\tlog->update.message = NULL;\n ++\t\tlog->value.update.old_hash = NULL;\n ++\t\tlog->value.update.new_hash = NULL;\n ++\t\tlog->value.update.message = NULL;\n +\t\tbreak;\n +\tcase REFTABLE_LOG_DELETION:\n +\t\tbreak;\n @@ refs/reftable-backend.c (new)\n +\n +\treftable_log_record_release(log);\n +\tlog->value_type = REFTABLE_LOG_UPDATE;\n -+\tlog->update.name =\n ++\tlog->value.update.name =\n +\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n -+\tlog->update.email =\n ++\tlog->value.update.email =\n +\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n -+\tlog->update.time = atol(split.date_begin);\n ++\tlog->value.update.time = atol(split.date_begin);\n +\tif (*split.tz_begin == '-') {\n +\t\tsign = -1;\n +\t\tsplit.tz_begin++;\n @@ refs/reftable-backend.c (new)\n +\t\tsplit.tz_begin++;\n +\t}\n +\n -+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n ++\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n +}\n +\n +static int has_suffix(struct strbuf *b, const char *suffix)\n @@ refs/reftable-backend.c (new)\n +\t\tlog->update_index = ts;\n +\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n +\t\tlog->refname = (char *)u->refname;\n -+\t\tlog->update.new_hash = u->new_oid.hash;\n -+\t\tlog->update.message = u->msg;\n ++\t\tlog->value.update.new_hash = u->new_oid.hash;\n ++\t\tlog->value.update.message = u->msg;\n +\n +\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n +\t\tif (err < 0)\n @@ refs/reftable-backend.c (new)\n +\n +\t\t/* XXX fold together with the old_id check below? */\n +\n -+\t\tlog->update.old_hash = old_id.hash;\n ++\t\tlog->value.update.old_hash = old_id.hash;\n +\t\tif (u->flags & REF_LOG_ONLY) {\n +\t\t\tcontinue;\n +\t\t}\n @@ refs/reftable-backend.c (new)\n +\t\tlog.update_index = ts;\n +\t\tlog.refname = (char *)arg->refnames->items[i].string;\n +\n -+\t\tlog.update.message = xstrdup(arg->logmsg);\n -+\t\tlog.update.new_hash = NULL;\n -+\t\tlog.update.old_hash = NULL;\n ++\t\tlog.value.update.message = xstrdup(arg->logmsg);\n ++\t\tlog.value.update.new_hash = NULL;\n ++\t\tlog.value.update.old_hash = NULL;\n +\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n +\t\t\t\t\t ¤t) == 0) {\n -+\t\t\tlog.update.old_hash =\n ++\t\t\tlog.value.update.old_hash =\n +\t\t\t\treftable_ref_record_val1(¤t);\n +\t\t}\n +\t\terr = reftable_writer_add_log(writer, &log);\n -+\t\tlog.update.old_hash = NULL;\n ++\t\tlog.value.update.old_hash = NULL;\n +\t\treftable_ref_record_release(¤t);\n +\n +\t\tclear_reftable_log_record(&log);\n @@ refs/reftable-backend.c (new)\n +\t\tfill_reftable_log_record(&log);\n +\t\tlog.refname = (char *)create->refname;\n +\t\tlog.update_index = ts;\n -+\t\tlog.update.message = (char *)create->logmsg;\n ++\t\tlog.value.update.message = (char *)create->logmsg;\n +\t\tif (refs_resolve_ref_unsafe(\n +\t\t\t (struct ref_store *)create->refs, create->refname,\n +\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n -+\t\t\tlog.update.old_hash = old_oid.hash;\n ++\t\t\tlog.value.update.old_hash = old_oid.hash;\n +\t\t}\n +\n +\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n +\t\t\t\t\t create->target, RESOLVE_REF_READING,\n +\t\t\t\t\t &new_oid, NULL) != NULL) {\n -+\t\t\tlog.update.new_hash = new_oid.hash;\n ++\t\t\tlog.value.update.new_hash = new_oid.hash;\n +\t\t}\n +\n -+\t\tif (log.update.old_hash != NULL ||\n -+\t\t log.update.new_hash != NULL) {\n ++\t\tif (log.value.update.old_hash != NULL ||\n ++\t\t log.value.update.new_hash != NULL) {\n +\t\t\terr = reftable_writer_add_log(writer, &log);\n +\t\t}\n +\t\tlog.refname = NULL;\n -+\t\tlog.update.message = NULL;\n -+\t\tlog.update.old_hash = NULL;\n -+\t\tlog.update.new_hash = NULL;\n ++\t\tlog.value.update.message = NULL;\n ++\t\tlog.value.update.old_hash = NULL;\n ++\t\tlog.value.update.new_hash = NULL;\n +\t\tclear_reftable_log_record(&log);\n +\t}\n +\treturn err;\n @@ refs/reftable-backend.c (new)\n +\n +\t\ttodo[0].refname = (char *)arg->oldname;\n +\t\ttodo[0].update_index = ts;\n -+\t\ttodo[0].update.message = (char *)arg->logmsg;\n -+\t\ttodo[0].update.old_hash = val1;\n -+\t\ttodo[0].update.new_hash = NULL;\n ++\t\ttodo[0].value.update.message = (char *)arg->logmsg;\n ++\t\ttodo[0].value.update.old_hash = val1;\n ++\t\ttodo[0].value.update.new_hash = NULL;\n +\n +\t\ttodo[1].refname = (char *)arg->newname;\n +\t\ttodo[1].update_index = ts;\n -+\t\ttodo[1].update.old_hash = NULL;\n -+\t\ttodo[1].update.new_hash = val1;\n -+\t\ttodo[1].update.message = (char *)arg->logmsg;\n ++\t\ttodo[1].value.update.old_hash = NULL;\n ++\t\ttodo[1].value.update.new_hash = val1;\n ++\t\ttodo[1].value.update.message = (char *)arg->logmsg;\n +\n +\t\terr = reftable_writer_add_logs(writer, todo, 2);\n +\n @@ refs/reftable-backend.c (new)\n +\n +\t\tfree(ri->last_name);\n +\t\tri->last_name = xstrdup(ri->log.refname);\n -+\t\toidread(&ri->oid, ri->log.update.new_hash);\n ++\t\toidread(&ri->oid, ri->log.value.update.new_hash);\n +\t\treturn ITER_OK;\n +\t}\n +}\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\toidread(&old_oid, log.update.old_hash);\n -+\t\toidread(&new_oid, log.update.new_hash);\n ++\t\toidread(&old_oid, log.value.update.old_hash);\n ++\t\toidread(&new_oid, log.value.update.new_hash);\n +\n -+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n ++\t\tfull_committer = fmt_ident(log.value.update.name,\n ++\t\t\t\t\t log.value.update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT,\n +\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n -+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n -+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n ++\t\terr = fn(&old_oid, &new_oid, full_committer,\n ++\t\t\t log.value.update.time, log.value.update.tz_offset,\n ++\t\t\t log.value.update.message, cb_data);\n +\t\tif (err)\n +\t\t\tbreak;\n +\t}\n @@ refs/reftable-backend.c (new)\n +\t\tstruct object_id new_oid;\n +\t\tconst char *full_committer = \"\";\n +\n -+\t\toidread(&old_oid, log->update.old_hash);\n -+\t\toidread(&new_oid, log->update.new_hash);\n ++\t\toidread(&old_oid, log->value.update.old_hash);\n ++\t\toidread(&new_oid, log->value.update.new_hash);\n +\n -+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n ++\t\tfull_committer = fmt_ident(log->value.update.name,\n ++\t\t\t\t\t log->value.update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n +\t\t\t\t\t IDENT_NO_DATE);\n -+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n -+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n ++\t\terr = fn(&old_oid, &new_oid, full_committer,\n ++\t\t\t log->value.update.time, log->value.update.tz_offset,\n ++\t\t\t log->value.update.message, cb_data);\n +\t\tif (err) {\n +\t\t\tbreak;\n +\t\t}\n @@ refs/reftable-backend.c (new)\n +\t\tstruct reftable_log_record *dest = &rewritten[i];\n +\n +\t\t*dest = logs[i];\n -+\t\toidread(&ooid, logs[i].update.old_hash);\n -+\t\toidread(&noid, logs[i].update.new_hash);\n -+\n -+\t\tif (should_prune_fn(&ooid, &noid, logs[i].update.email,\n -+\t\t\t\t (timestamp_t)logs[i].update.time,\n -+\t\t\t\t logs[i].update.tz_offset,\n -+\t\t\t\t logs[i].update.message, policy_cb_data)) {\n ++\t\toidread(&ooid, logs[i].value.update.old_hash);\n ++\t\toidread(&noid, logs[i].value.update.new_hash);\n ++\n ++\t\tif (should_prune_fn(&ooid, &noid, logs[i].value.update.email,\n ++\t\t\t\t (timestamp_t)logs[i].value.update.time,\n ++\t\t\t\t logs[i].value.update.tz_offset,\n ++\t\t\t\t logs[i].value.update.message,\n ++\t\t\t\t policy_cb_data)) {\n +\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n +\t\t} else {\n +\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n +\t\t\t last_hash != NULL) {\n -+\t\t\t\tdest->update.old_hash = last_hash;\n ++\t\t\t\tdest->value.update.old_hash = last_hash;\n +\t\t\t}\n -+\t\t\tlast_hash = logs[i].update.new_hash;\n ++\t\t\tlast_hash = logs[i].value.update.new_hash;\n +\t\t}\n +\t}\n +\n 21: 79c4b90d0f4 = 21: 460658a13cb git-prompt: prepare for reftable refs backend\n 22: 1f48091e549 = 22: b305fcd49f1 Add \"test-tool dump-reftable\" command.\n 23: 64dc87746db = 23: a44956d7b08 t1301: document what needs to be done for reftable\n 24: c248b69c6e3 = 24: f0b7b1e2380 t1401,t2011: parameterize HEAD.lock for REFFILES\n 25: 9f1b792459e = 25: 3110d6e6758 t1404: annotate test cases with REFFILES\n\n-- \ngitgitgadget\n"},{"id":"432981","messageId":"97aba3ae406859231ca4d978becf23497511da87.1629207607.git.gitgitgadget@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v3 02/25] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:39:44Z","receivedAt":"2021-08-17T13:44:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 2167796ff2a..c2f03f6018e 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -425,6 +425,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"432983","messageId":"CAFQ2z_O1mJEqnxow-SLytiPwh_Pgnsupdi5ORgE4Q6KULmuLoA@mail.gmail.com","threadId":"56133","inReplyTo":"YRresPEnuv9EwBgS@carlos-mbp.lan","subject":"Re: [PATCH v2 07/25] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-17T13:44:16Z","receivedAt":"2021-08-17T13:44:35Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Aug 16, 2021 at 11:54 PM Carlo Marcelo Arenas Belón\n wrote:\n>\n> On Mon, Aug 16, 2021 at 08:16:58PM +0000, Han-Wen Nienhuys via GitGitGadget wrote:\n> > From: Han-Wen Nienhuys \n> ...\n> > + union {\n> > + struct {\n> > + uint8_t *new_hash;\n> > + uint8_t *old_hash;\n> > + char *name;\n> > + char *email;\n> > + uint64_t time;\n> > + int16_t tz_offset;\n> > + char *message;\n> > + } update;\n> > + };\n>\n> the use of an anonymous union here (which requires C11) could be made\n> C89 compatible without any impact AFAIK by doing instead:\n\nThanks for the pointer. I opted to do it the C89 way instead, for\nconsistency with reftable_ref_record.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"432991","messageId":"CAFQ2z_N0s3-95CxVtj+HQu2AUiZoRK1N3gVAFg0ewOviQWmofA@mail.gmail.com","threadId":"56133","inReplyTo":"xmqqr1etqfnp.fsf@gitster.g","subject":"Re: [PATCH v2 00/25] Support reftable ref backend for Git","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-17T16:38:28Z","receivedAt":"2021-08-17T16:38:44Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Aug 16, 2021 at 10:49 PM Junio C Hamano wrote:\n>\n> \"Han-Wen Nienhuys via GitGitGadget\" writes:\n>\n> > This continues the work in https://github.com/git/git/pull/847, which the\n> > gitgitgadget erroneously closed.\n>\n> > Changes relative to last series (version 20 Jul 2021, tip: 4a5891f )\n> >\n> > * fix HTTP serving\n> > * fix t1501 (use absolute paths.)\n> > * fix git-new-workdir\n> > * fixes for FreeBSD (thanks, carenas@gmail.com!)\n>\n> Thanks, will take a look later after the release is done; nice to\n> see these fix-up changes already applied in the series.\n\nYou're welcome.\n\nSlight tangent: for the series, the bottom-most commit (which\nintroduces the hash identifiers) could be a separate topic which\ngraduates sooner.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"433058","messageId":"e1959ed9-beed-f110-c9a8-da8ed352dcda@iee.email","threadId":"56133","inReplyTo":"d92338467d66fcfedd57f209c97a798e9920d1e5.1629207607.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 08/25] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-08-18T10:14:08Z","receivedAt":"2021-08-18T10:14:18Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 17/08/2021 14:39, Han-Wen Nienhuys via GitGitGadget wrote:\n> From: Han-Wen Nienhuys \n>\n> This will be needed for reading reflog blocks in reftable.\n\nHow large might the reftable become? In particular will it exceed the\n32bit Long limit on Windows?\n\nI ask as the Zlib library is one of (among many) the constraints on\nbeating the 4GB [backward compatibility] size limit from 32 bit Windows.\n\n>\n> Helped-by: Carlo Marcelo Arenas Belón \n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> Makefile | 7 +++\n> ci/lib.sh | 1 +\n> compat/.gitattributes | 1 +\n> compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n> config.mak.uname | 1 +\n> configure.ac | 13 ++++++\n> 6 files changed, 115 insertions(+)\n> create mode 100644 compat/.gitattributes\n> create mode 100644 compat/zlib-uncompress2.c\n>\n> diff --git a/Makefile b/Makefile\n> index e98d8ed17cf..16c883978d4 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -256,6 +256,8 @@ all::\n> #\n> # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n> #\n> +# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n> +#\n> # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n> # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n> #\n> @@ -1738,6 +1740,11 @@ ifdef NO_DEFLATE_BOUND\n> \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n> endif\n> \n> +ifdef NO_UNCOMPRESS2\n> +\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n> +\tREFTABLE_OBJS += compat/zlib-uncompress2.o\n> +endif\n> +\n> ifdef NO_POSIX_GOODIES\n> \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n> endif\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 476c3f369f5..5711c63979d 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -224,6 +224,7 @@ linux-gcc-default)\n> \t;;\n> Linux32)\n> \tCC=gcc\n> +\tMAKEFLAGS=\"$MAKEFLAGS NO_UNCOMPRESS2=1\"\n> \t;;\n> linux-musl)\n> \tCC=gcc\n> diff --git a/compat/.gitattributes b/compat/.gitattributes\n> new file mode 100644\n> index 00000000000..40dbfb170da\n> --- /dev/null\n> +++ b/compat/.gitattributes\n> @@ -0,0 +1 @@\n> +/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\n> diff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\n> new file mode 100644\n> index 00000000000..6893bb469ce\n> --- /dev/null\n> +++ b/compat/zlib-uncompress2.c\n> @@ -0,0 +1,92 @@\n> +/* taken from zlib's uncompr.c\n> +\n> + commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n> + Author: Mark Adler \n> + Date: Sun Jan 15 09:18:46 2017 -0800\n> +\n> + zlib 1.2.11\n> +\n> +*/\n> +\n> +/*\n> + * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n> + * For conditions of distribution and use, see copyright notice in zlib.h\n> + */\n> +\n> +#include \n> +\n> +/* clang-format off */\n> +\n> +/* ===========================================================================\n> + Decompresses the source buffer into the destination buffer. *sourceLen is\n> + the byte length of the source buffer. Upon entry, *destLen is the total size\n> + of the destination buffer, which must be large enough to hold the entire\n> + uncompressed data. (The size of the uncompressed data must have been saved\n> + previously by the compressor and transmitted to the decompressor by some\n> + mechanism outside the scope of this compression library.) Upon exit,\n> + *destLen is the size of the decompressed data and *sourceLen is the number\n> + of source bytes consumed. Upon return, source + *sourceLen points to the\n> + first unused input byte.\n> +\n> + uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n> + memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n> + Z_DATA_ERROR if the input data was corrupted, including if the input data is\n> + an incomplete zlib stream.\n> +*/\n> +int ZEXPORT uncompress2 (\n> + Bytef *dest,\n> + uLongf *destLen,\n> + const Bytef *source,\n> + uLong *sourceLen) {\n\nLength is limited to 4GB on Windows (the pointer itself will be\nimplicitly size_t, but ...)\n\n> + z_stream stream;\n> + int err;\n> + const uInt max = (uInt)-1;\n> + uLong len, left;\n> + Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n> +\n> + len = *sourceLen;\n> + if (*destLen) {\n> + left = *destLen;\n> + *destLen = 0;\n> + }\n> + else {\n> + left = 1;\n> + dest = buf;\n> + }\n> +\n> + stream.next_in = (z_const Bytef *)source;\n> + stream.avail_in = 0;\n> + stream.zalloc = (alloc_func)0;\n> + stream.zfree = (free_func)0;\n> + stream.opaque = (voidpf)0;\n> +\n> + err = inflateInit(&stream);\n> + if (err != Z_OK) return err;\n> +\n> + stream.next_out = dest;\n> + stream.avail_out = 0;\n> +\n> + do {\n> + if (stream.avail_out == 0) {\n> + stream.avail_out = left > (uLong)max ? max : (uInt)left;\n> + left -= stream.avail_out;\n> + }\n> + if (stream.avail_in == 0) {\n> + stream.avail_in = len > (uLong)max ? max : (uInt)len;\n> + len -= stream.avail_in;\n> + }\n> + err = inflate(&stream, Z_NO_FLUSH);\n> + } while (err == Z_OK);\n> +\n> + *sourceLen -= len + stream.avail_in;\n> + if (dest != buf)\n> + *destLen = stream.total_out;\n> + else if (stream.total_out && err == Z_BUF_ERROR)\n> + left = 1;\n> +\n> + inflateEnd(&stream);\n> + return err == Z_STREAM_END ? Z_OK :\n> + err == Z_NEED_DICT ? Z_DATA_ERROR :\n> + err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n> + err;\n> +}\n> diff --git a/config.mak.uname b/config.mak.uname\n> index 69413fb3dc0..61e11550b1f 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -256,6 +256,7 @@ ifeq ($(uname_S),FreeBSD)\n> \tFILENO_IS_A_MACRO = UnfortunatelyYes\n> endif\n> ifeq ($(uname_S),OpenBSD)\n> +\tNO_UNCOMPRESS2 = YesPlease\n> \tNO_STRCASESTR = YesPlease\n> \tNO_MEMMEM = YesPlease\n> \tUSE_ST_TIMESPEC = YesPlease\n> diff --git a/configure.ac b/configure.ac\n> index 031e8d3fee8..c3a913103d0 100644\n> --- a/configure.ac\n> +++ b/configure.ac\n> @@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n> \tNO_DEFLATE_BOUND=yes])\n> LIBS=\"$old_LIBS\"\n> \n> +AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n> +AC_LANG_PROGRAM([#include ],\n> + [uncompress2(NULL,NULL,NULL,NULL);])])\n> +AC_MSG_CHECKING([for uncompress2 in -lz])\n> +old_LIBS=\"$LIBS\"\n> +LIBS=\"$LIBS -lz\"\n> +AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n> +\t[AC_MSG_RESULT([yes])],\n> +\t[AC_MSG_RESULT([no])\n> +\tNO_UNCOMPRESS2=yes])\n> +LIBS=\"$old_LIBS\"\n> +\n> GIT_UNSTASH_FLAGS($ZLIB_PATH)\n> \n> GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n> +GIT_CONF_SUBST([NO_UNCOMPRESS2])\n> \n> #\n> # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n--\nPhilip\n"},{"id":"433062","messageId":"CAFQ2z_P1EnNYPxiTZ8GeTTVsJaa0S0vz5M4kqu7Kf3yGbUt0kQ@mail.gmail.com","threadId":"56133","inReplyTo":"e1959ed9-beed-f110-c9a8-da8ed352dcda@iee.email","subject":"Re: [PATCH v3 08/25] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-18T10:39:01Z","receivedAt":"2021-08-18T10:39:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Aug 18, 2021 at 12:14 PM Philip Oakley wrote:\n>\n> On 17/08/2021 14:39, Han-Wen Nienhuys via GitGitGadget wrote:\n> > From: Han-Wen Nienhuys \n> >\n> > This will be needed for reading reflog blocks in reftable.\n>\n> How large might the reftable become? In particular will it exceed the\n> 32bit Long limit on Windows?\n\nA reftable file is organized as a sequence of blocks. Each block has\nmax size 2^24, and the zlib compression is done per-block. So zlib\nshould not introduce a new 4G limit. The intra-file offsets for the\nblocks themselves are encoded either as varint (index values) or as\nuint64 (file footer), so reftable files beyond 4G should be no\nproblem.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"433066","messageId":"916368b8-7c6f-09df-3dd8-03a4f99f9d1a@iee.email","threadId":"56133","inReplyTo":"CAFQ2z_P1EnNYPxiTZ8GeTTVsJaa0S0vz5M4kqu7Kf3yGbUt0kQ@mail.gmail.com","subject":"Re: [PATCH v3 08/25] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-08-18T11:53:37Z","receivedAt":"2021-08-18T11:53:42Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 18/08/2021 11:39, Han-Wen Nienhuys wrote:\n> On Wed, Aug 18, 2021 at 12:14 PM Philip Oakley wrote:\n>> On 17/08/2021 14:39, Han-Wen Nienhuys via GitGitGadget wrote:\n>>> From: Han-Wen Nienhuys \n>>>\n>>> This will be needed for reading reflog blocks in reftable.\n>> How large might the reftable become? In particular will it exceed the\n>> 32bit Long limit on Windows?\n> A reftable file is organized as a sequence of blocks. Each block has\n> max size 2^24, and the zlib compression is done per-block. So zlib\n> should not introduce a new 4G limit. The intra-file offsets for the\n> blocks themselves are encoded either as varint (index values) or as\n> uint64 (file footer), so reftable files beyond 4G should be no\n> problem.\n>\nThanks, that's good to know for planning any Git upgrade to cope with\n4GB file sizes on Windows.\n\nPhilip\n"},{"id":"433356","messageId":"877dgch4rn.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/25] Support reftable ref backend for Git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T09:08:14Z","receivedAt":"2021-08-23T09:47:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 17 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> This continues the work in https://github.com/git/git/pull/847, which the\n> gitgitgadget erroneously closed.\n>\n> Changes relative to last series (version 20 Jul 2021, tip: 4a5891f )\n>\n> * fix HTTP serving\n> * fix t1501 (use absolute paths.)\n> * fix git-new-workdir\n> * fixes for FreeBSD (thanks, carenas@gmail.com!)\n>\n\nWhat's the overall status of this topic vis-a-vis whether it's being\nproposed to be integrated or is in more RFC form?\n\nSince we discussed the non-working GIT_TEST_REFTABLE=true in [1] & [2]\nthere's been fixes to the REFFILES for the tests which has already\nlanded as c9780bb2ca8 (Merge branch 'hn/prep-tests-for-reftable',\n2021-07-13).\n\nBut e.g. running t1410-reflog.sh reveals an assert failure() in this\nlibrary's own code:\n\n git: reftable/record.c:1075: reftable_record_copy_from: Assertion\n `src->ops->type == rec->ops->type' failed.\n\n(A *lot* of things in the test suite still fail, I didn't look in any\ndetail. Just highlighting that it's not just remaining failures due to\nhardcoding of .git/refs/*).\n\nI took your reply in [2] to mean that you were on board with the plan I\nproposed in [1], but aside from the limited REFFILES fixes the structure\nof what's here seems relatively unchanged.\n\nSo is this series still considered RFC-esque, or what's the plan to move\nthis forward at this point?\n\n1. https://lore.kernel.org/git/87h7jqz7k5.fsf@evledraar.gmail.com\n2. https://lore.kernel.org/git/CAFQ2z_P8vgY0RRT+XSH9K3VDQt39FLqXx6qfeZqaZPkwhq1w+A@mail.gmail.com\n"},{"id":"433357","messageId":"874kbgh4qt.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"cad3c4835a2945cfecc66da161fefa53d9c19574.1629207607.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/25] hash.h: provide constants for the hash IDs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T09:47:19Z","receivedAt":"2021-08-23T09:47:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 17 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> @@ -164,7 +164,6 @@ static void git_hash_unknown_final_oid(struct object_id *oid, git_hash_ctx *ctx)\n> \tBUG(\"trying to finalize unknown hash\");\n> }\n> \n> -\n> const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n> \t{\n> \t\tNULL,\n\nNit: stray whitespace change that snuck in?\n"},{"id":"433358","messageId":"871r6kh3m8.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"3d57f7c443082fd2a7f01aee003a9cd3ca2dd910.1629207607.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 20/25] refs: RFC: Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T09:50:18Z","receivedAt":"2021-08-23T10:12:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 17 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> [...]\n> +\t\tif (len == cap) {\n> +\t\t\tcap = 2 * cap + 1;\n> +\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n> +\t\t}\n> +\n> +\t\tlogs[len++] = log;\n> [...]\n> +\t\tif (logs_len >= logs_cap) {\n> +\t\t\tint new_cap = logs_cap * 2 + 1;\n> +\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n> +\t\t\tlogs_cap = new_cap;\n> +\t\t}\n> +\t\tlogs[logs_len++] = log;\n> +\t}\n\nElsewhere in this series we use the ARRAY_SIZE() macro from\ngit-compat-util.h, can't we also use REALLOC_ARRAY() from the same\nheader here?\n\n> [...]\n> +\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n> +\t\t/* XXX - skip writing records that were not changed. */\n> +\t\terr = reftable_addition_commit(add);\n> +\t} else {\n> +\t\t/* XXX - print something */\n> +\t}\n\nAren't these comments covered by some of the failing tests under\nGIT_TEST_REFTABLE=true? I.e. what I mentioned\nhttps://lore.kernel.org/git/877dgch4rn.fsf@evledraar.gmail.com/\n\n> [...]\n> +\tif (err < 0) {\n> +\t\terrno = reftable_error_to_errno(err);\n> +\t\terr = -1;\n> +\t\tgoto done;\n> +\t}\n\nIn your proposed fixup for the merger of our topics in\nhttps://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\nyou have the call to reftable_error_to_errno() here deleted, so isn't\nthis also redundant at this point (and then the\nreftable_error_to_errno() function can be deleted), or is this errno\nsetting still needed with some of my changes?\n\n> [...]\n> +git_init () {\n> +\tgit init -b primary \"$@\"\n> +}\n\nCan't later tests just use \"main\" instead of primary with a\nGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main, then we can drop this\n\"git_init\" wrapper?\n\n> +initialize () {\n> +\trm -rf .git &&\n\nShould instead set up a test_when_finished \"rm -rf .git\" ?\n\n> +\t(GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n\nThis export before calling git_init can surely go away if git_init goes,\nbut alo why export beforehand here, but in later tests in this file we\njust do a plain:\n\n (GIT_TEST_REFTABLE=1 git_init) &&\n\n?\n\n> +\tmv .git/hooks .git/hooks-disabled\n\nIs this \"mv\" cargo-culted from what test_create_repo() used to do before\nmy-f0d4d398e28 (test-lib: split up and deprecate test_create_repo(),\n2021-05-10)? In any case templated hooks are disabled by default (named\n*.sample), so I don't think this is needed.\n\n> +\techo hoi >> file.t &&\n\nNit: >>file.t, not \">> file.t\".\n\n> +\tgit show-ref | sed s/before/after/g > expected &&\n\nDon't have \"git\" on the LHS of a pipe, it'll hide a segfault. Should use\na temporary file. Also \"s/ > />/g\" like above (and in some places below,\nwill stop noting it...).\n\n> +test_expect_success 'SHA256 support, env' '\n> +\trm -rf .git &&\n> +\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n> +\t(GIT_TEST_REFTABLE=1 git_init) &&\n> +\tmv .git/hooks .git/hooks-disabled &&\n\nComments about this .git/hooks-disabled not being needed aside, this\nseems to be duplicating the initialize() function. I.e. should we not\nskip the \"rm -rf\" and \"mv\" here, and just set GIT_DEFAULT_HASH=sha256\nand call initialize?\n\n(Better yet, if we get rid of that \"git init\" wrapper as I noted above,\nthis can just be an argument to \"git init\", no?\n\n> [...]\n> +\trm -rf .git &&\n> +\t(GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n> +\tmv .git/hooks .git/hooks-disabled &&\n\nditto.\n\n> +\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n\nAlways \"test_must_fail git\", not \"! git\".\n\n> [...]\n> +test_expect_success 'clone calls transaction_initial_commit' '\n> +\ttest_commit message1 file1 &&\n> +\tgit clone . cloned &&\n> +\t(test -f cloned/file1 || echo \"Fixme.\")\n\nSo this test really tests nothing much, and we should skip the \"Fixme\"\nhere and have this be test_expect_failure() or something instead?\n\n> [...]\n> +\tgit show-ref | cut -f2 -d\" \" >actual &&\n\nGit on LHS of a pipe again.\n\n> +# This matches show-ref's output\n> +print_ref() {\n> +\techo \"$(git rev-parse \"$1\") $1\"\n> +}\n> +\n> +test_expect_success 'peeled tags are stored' '\n> +\tinitialize &&\n> +\ttest_commit file &&\n> +\tgit tag -m \"annotated tag\" test_tag HEAD &&\n> +\t{\n> +\t\tprint_ref \"refs/heads/primary\" &&\n> +\t\tprint_ref \"refs/tags/file\" &&\n> +\t\tprint_ref \"refs/tags/test_tag\" &&\n> +\t\tprint_ref \"refs/tags/test_tag^{}\"\n> +\t} >expect &&\n\nMaybe I'm missing something, but wouldn't this print_ref() helper be\nbetter as a \"git for-each-ref --format\" of some sort?\n\n> [...]\n> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> +start_httpd\n> +\n> +REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n\nLet's split these httpd-needing tests into another test file, see\nhttps://lore.kernel.org/git/87bl753i2p.fsf@evledraar.gmail.com/ for why.\n\n> [...]\n> +if test_have_prereq !REFFILES\n> +then\n> + skip_all='skipping pack-refs tests; need files backend'\n> + test_done\n> +fi\n\nIndent with spaces?\n\n> +if test_have_prereq !REFFILES\n> +then\n> + skip_all='skipping tests; incompatible with reftable'\n> + test_done\n> +fi\n> +\n\nDitto.\n\n(In general for issues noted above, I saw many more of some of them,\nincluding but not limited to this space formatting issue, but elided the\npatch after the first occurrences).\n"},{"id":"433399","messageId":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com","subject":"[PATCH v4 00/28] Support reftable ref backend for Git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:11Z","receivedAt":"2021-08-23T12:12:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This is a version of the reftable series queued on top of my\njust-re-rolled fixes to the refs APIs, which it can make use of. For\nthe base topics see:\n\n https://lore.kernel.org/git/cover-v5-00.13-00000000000-20210823T113115Z-avarab@gmail.com/\n https://lore.kernel.org/git/cover-v10-0.8-00000000000-20210823T114712Z-avarab@gmail.com/\n\nFor Han-Wen's v3 of this see:\nhttps://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\n\nI've got no desire to take over the reftable topic in its entirety,\nbut think given the rationale in\nhttps://lore.kernel.org/git/877dgch4rn.fsf@evledraar.gmail.com/\n(summarized in\nhttps://lore.kernel.org/git/87y28sfokk.fsf@evledraar.gmail.com/) that\nhaving the refs API fixes I noted above wait on the still-unstable\nreftable doesn't make sense.\n\nI think that plan came about due to my non-responsiveness for the past\ncouple of weeks (I was on vacation).\n\nSo I'm submitting this so Junio has something to pick up and put in\nhn/reftable which compiles and passes tests, this should clear up the\noutstanding issues in the \"seen\" branch.\n\nAs the range-diff shows the only changes are the addition of three new\npatches at the end. Those patches clearly need to be squashed into\ntheir respective preceding commits.\n\nI'll let Han-Wen deal with that squashing in a presumed future v5 of\nthis, assuming of course that Junio's happy with the plan of basing\nhn/reftable on the refs API fixes above.\n\nI'm not sure that the fix I have in 27/28 is the right one, perhaps\nwe've already got information about what the tip OID of the refname is\nat that point in git_reftable_reflog_expire() via some API I missed,\nbut that fix works, and is clearly more correct than the outstanding\nsegfault.\n\nHan-Wen Nienhuys (24):\n hash.h: provide constants for the hash IDs\n init-db: set the_repository->hash_algo early on\n reftable: RFC: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n Provide zlib's uncompress2 from compat/zlib-compat.c\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n refs: RFC: Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n t1301: document what needs to be done for reftable\n t1401,t2011: parameterize HEAD.lock for REFFILES\n t1404: annotate test cases with REFFILES\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\nÆvar Arnfjörð Bjarmason (3):\n reftable: fixup for new base topic 1/3\n reftable: fixup for new base topic 2/3\n reftable: fixup for new base topic 3/3\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 54 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 60 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +\n config.mak.uname | 3 +-\n configure.ac | 13 +\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n contrib/workdir/git-new-workdir | 2 +-\n hash.h | 6 +\n object-file.c | 7 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1687 +++++++++++++++++\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 +\n reftable/block.c | 448 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 120 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 107 ++\n reftable/error.c | 41 +\n reftable/generic.c | 169 ++\n reftable/generic.h | 32 +\n reftable/iter.c | 194 ++\n reftable/iter.h | 69 +\n reftable/merged.c | 362 ++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 292 +++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 801 ++++++++\n reftable/reader.h | 66 +\n reftable/readwrite_test.c | 652 +++++++\n reftable/record.c | 1212 ++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 412 ++++\n reftable/refname.c | 209 ++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 47 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 101 +\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 128 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 148 ++\n reftable/reftable.c | 115 ++\n reftable/stack.c | 1396 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 948 +++++++++\n reftable/system.h | 24 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 690 +++++++\n reftable/writer.h | 50 +\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 316 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1301-shared-repo.sh | 8 +-\n t/t1401-symbolic-ref.sh | 11 +-\n t/t1404-update-ref-errors.sh | 56 +-\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t2011-checkout-invalid-head.sh | 11 +-\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 94 files changed, 13174 insertions(+), 62 deletions(-)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/readwrite_test.c\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\nRange-diff against v3:\n 1: cad3c4835a2 = 1: 295ccff312f hash.h: provide constants for the hash IDs\n 2: 97aba3ae406 = 2: 2fb0fb20282 init-db: set the_repository->hash_algo early on\n 3: fca162dd676 = 3: 6cf0b244fc5 reftable: RFC: add LICENSE\n 4: 44923074a5a = 4: a0686356c11 reftable: add error related functionality\n 5: 2f2beb70078 = 5: 23ceb2e5723 reftable: utility functions\n 6: 79d4a23764d = 6: 9429aaf7c76 reftable: add blocksource, an abstraction for random access reads\n 7: 2c143c75cd5 = 7: f4cebccb9e0 reftable: (de)serialization for the polymorphic record type.\n 8: d92338467d6 = 8: e2f2729443b Provide zlib's uncompress2 from compat/zlib-compat.c\n 9: 74ac0b135fa = 9: 9d468fbdd07 reftable: reading/writing blocks\n10: a2219670079 = 10: 7d8b8e3b76a reftable: a generic binary tree implementation\n11: ac3063fd08c = 11: 7ec422318eb reftable: write reftable files\n12: e0f3476555e = 12: df4fae30ceb reftable: generic interface to tables\n13: f1fe338f67e = 13: 7e8a613206f reftable: read reftable files\n14: 7da9a85c1db = 14: ebe55aa6295 reftable: reftable file level tests\n15: 953590034a8 = 15: a9880fc1a10 reftable: add a heap-based priority queue for reftable records\n16: 64fe3c8edc5 = 16: c073be56a6b reftable: add merged table view\n17: acb0d435a20 = 17: 092b9715672 reftable: implement refname validation\n18: 764d45fccd4 = 18: 374c22efbd9 reftable: implement stack, a mutable database of reftable files.\n19: 88dbab683dc = 19: 007c9359f42 reftable: add dump utility\n20: 3d57f7c4430 = 20: 8f948c0b257 refs: RFC: Reftable support for git-core\n21: 460658a13cb = 21: 443bdebfb5d git-prompt: prepare for reftable refs backend\n22: b305fcd49f1 = 22: f76e7a7bd10 Add \"test-tool dump-reftable\" command.\n23: a44956d7b08 = 23: 6b62a8c52ae t1301: document what needs to be done for reftable\n24: f0b7b1e2380 = 24: c926f1526aa t1401,t2011: parameterize HEAD.lock for REFFILES\n25: 3110d6e6758 = 25: 0b3db9fc2d6 t1404: annotate test cases with REFFILES\n -: ----------- > 26: dde80a982c1 reftable: fixup for new base topic 1/3\n -: ----------- > 27: c4f9fb42d9e reftable: fixup for new base topic 2/3\n -: ----------- > 28: c863b03201c reftable: fixup for new base topic 3/3\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433397","messageId":"patch-v4-01.28-295ccff312f-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 01/28] hash.h: provide constants for the hash IDs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:12Z","receivedAt":"2021-08-23T12:12:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will simplify referencing them from code that is not deeply integrated with\nGit, in particular, the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n hash.h | 6 ++++++\n object-file.c | 7 ++-----\n 2 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9e25c40e9ac..5d40368f18a 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -95,12 +95,18 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n /* Number of algorithms supported (including unknown). */\n #define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n+/* \"sha1\", big-endian */\n+#define GIT_SHA1_FORMAT_ID 0x73686131\n+\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* \"s256\", big-endian */\n+#define GIT_SHA256_FORMAT_ID 0x73323536\n+\n /* The length in bytes and in hex digits of an object name (SHA-256 value). */\n #define GIT_SHA256_RAWSZ 32\n #define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\ndiff --git a/object-file.c b/object-file.c\nindex 9b318eecb19..e8ca385a09c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -164,7 +164,6 @@ static void git_hash_unknown_final_oid(struct object_id *oid, git_hash_ctx *ctx)\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n-\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\tNULL,\n@@ -183,8 +182,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha1\",\n-\t\t/* \"sha1\", big-endian */\n-\t\t0x73686131,\n+\t\tGIT_SHA1_FORMAT_ID,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n \t\tGIT_SHA1_BLKSZ,\n@@ -199,8 +197,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha256\",\n-\t\t/* \"s256\", big-endian */\n-\t\t0x73323536,\n+\t\tGIT_SHA256_FORMAT_ID,\n \t\tGIT_SHA256_RAWSZ,\n \t\tGIT_SHA256_HEXSZ,\n \t\tGIT_SHA256_BLKSZ,\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433398","messageId":"patch-v4-02.28-2fb0fb20282-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 02/28] init-db: set the_repository->hash_algo early on","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:13Z","receivedAt":"2021-08-23T12:13:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 2167796ff2a..c2f03f6018e 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -425,6 +425,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433400","messageId":"patch-v4-03.28-6cf0b244fc5-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 03/28] reftable: RFC: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:14Z","receivedAt":"2021-08-23T12:13:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe objective of this code is to be usable as a C library, so it can be reused\nin libgit2.\n\nThis is currently using a BSD license as it is the liberal license I could find,\nbut this could be changed to whatever fits the stated goal above.\n\nThis code is currently imported from github.com/hanwen/reftable. Once this code\nlands in git.git, the C code will be removed from github.com/hanwen/reftable,\nand the git.git code will be the source of truth.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 00000000000..402e0f9356b\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433401","messageId":"patch-v4-04.28-a0686356c11-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 04/28] reftable: add error related functionality","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:15Z","receivedAt":"2021-08-23T12:13:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 00000000000..f6f16def921\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 00000000000..6f89bedf1a5\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433403","messageId":"patch-v4-05.28-23ceb2e5723-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 05/28] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:16Z","receivedAt":"2021-08-23T12:13:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 24 ++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 545 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d7..addf2813086 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -743,6 +743,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -821,6 +822,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1195,7 +1198,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2442,7 +2445,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2458,6 +2469,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2608,6 +2621,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2895,7 +2914,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3225,7 +3244,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 171b4124afe..c2bf5bdffc6 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -640,6 +640,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -662,7 +668,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -902,11 +908,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 00000000000..f761e48028c\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (!a) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 00000000000..096b36862b9\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 00000000000..1fcd2297256\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(!out[2]);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!out[1]);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 00000000000..bd0a02d3f68\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 00000000000..5f2185f1f34\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 00000000000..5e7698ae654\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 00000000000..4f62827b83b\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,24 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+/* This header glues the reftable library to the rest of Git */\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hash.h\" /* hash ID, sizes.*/\n+#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+\n+#include \n+\n+struct strbuf;\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 00000000000..84ac972cad0\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_FORMAT_ID));\n+}\n+\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add(b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 00000000000..774cb275bf6\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 00000000000..3b58e423e7b\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53a..f7c888ffda7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -53,13 +53,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f5228508..25f77469146 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -49,6 +49,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 00000000000..0ed14971a58\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433402","messageId":"patch-v4-06.28-9429aaf7c76-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 06/28] reftable: add blocksource, an abstraction for random access reads","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:17Z","receivedAt":"2021-08-23T12:13:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex addf2813086..86ba58cf140 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2447,6 +2447,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 00000000000..0044eecd9aa\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(!bs->ops);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(!bs->ops);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 00000000000..072e2727ad2\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 00000000000..5aa3990a573\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433404","messageId":"patch-v4-07.28-f4cebccb9e0-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 07/28] reftable: (de)serialization for the polymorphic record type.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:18Z","receivedAt":"2021-08-23T12:13:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1212 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 412 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1901 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 86ba58cf140..e98d8ed17cf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2449,7 +2449,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 00000000000..5eee72c4c11\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 00000000000..6a5dac32dc6\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = rec;\n+\tconst struct reftable_ref_record *src = src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release(rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tobj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));\n+\tCOPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->value.update.name,\n+\t\t log->value.update.email, log->value.update.time,\n+\t\t log->value.update.tz_offset);\n+\t\thex_format(hex, log->value.update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->value.update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->value.update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->value.update.email) {\n+\t\t\tdst->value.update.email =\n+\t\t\t\txstrdup(dst->value.update.email);\n+\t\t}\n+\t\tif (dst->value.update.name) {\n+\t\t\tdst->value.update.name =\n+\t\t\t\txstrdup(dst->value.update.name);\n+\t\t}\n+\t\tif (dst->value.update.message) {\n+\t\t\tdst->value.update.message =\n+\t\t\t\txstrdup(dst->value.update.message);\n+\t\t}\n+\n+\t\tif (dst->value.update.new_hash) {\n+\t\t\tdst->value.update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->value.update.new_hash,\n+\t\t\t src->value.update.new_hash, hash_size);\n+\t\t}\n+\t\tif (dst->value.update.old_hash) {\n+\t\t\tdst->value.update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->value.update.old_hash,\n+\t\t\t src->value.update.old_hash, hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->value.update.new_hash);\n+\t\treftable_free(r->value.update.old_hash);\n+\t\treftable_free(r->value.update.name);\n+\t\treftable_free(r->value.update.email);\n+\t\treftable_free(r->value.update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_log_record *r = rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->value.update.old_hash;\n+\tnewh = r->value.update.new_hash;\n+\tif (!oldh) {\n+\t\toldh = zero;\n+\t}\n+\tif (!newh) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->value.update.name ? r->value.update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->value.update.email ? r->value.update.email : \"\",\n+\t\t\t s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->value.update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->value.update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(\n+\t\tr->value.update.message ? r->value.update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->value.update.old_hash);\n+\t\t\tFREE_AND_NULL(r->value.update.new_hash);\n+\t\t\tFREE_AND_NULL(r->value.update.message);\n+\t\t\tFREE_AND_NULL(r->value.update.email);\n+\t\t\tFREE_AND_NULL(r->value.update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->value.update.old_hash =\n+\t\treftable_realloc(r->value.update.old_hash, hash_size);\n+\tr->value.update.new_hash =\n+\t\treftable_realloc(r->value.update.new_hash, hash_size);\n+\n+\tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.name =\n+\t\treftable_realloc(r->value.update.name, dest.len + 1);\n+\tmemcpy(r->value.update.name, dest.buf, dest.len);\n+\tr->value.update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.email =\n+\t\treftable_realloc(r->value.update.email, dest.len + 1);\n+\tmemcpy(r->value.update.email, dest.buf, dest.len);\n+\tr->value.update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->value.update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->value.update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->value.update.message =\n+\t\treftable_realloc(r->value.update.message, dest.len + 1);\n+\tmemcpy(r->value.update.message, dest.buf, dest.len);\n+\tr->value.update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (!a)\n+\t\ta = empty;\n+\n+\tif (!b)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (!a)\n+\t\ta = zero;\n+\n+\tif (!b)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->value.update.name, b->value.update.name) &&\n+\t\t a->value.update.time == b->value.update.time &&\n+\t\t a->value.update.tz_offset == b->value.update.tz_offset &&\n+\t\t null_streq(a->value.update.email,\n+\t\t\t\t b->value.update.email) &&\n+\t\t null_streq(a->value.update.message,\n+\t\t\t\t b->value.update.message) &&\n+\t\t zero_hash_eq(a->value.update.old_hash,\n+\t\t\t\t b->value.update.old_hash, hash_size) &&\n+\t\t zero_hash_eq(a->value.update.new_hash,\n+\t\t\t\t b->value.update.new_hash, hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_index_record *rec = r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = rec;\n+\tconst struct reftable_index_record *src = src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a && b)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tconst struct reftable_log_record *la = a;\n+\tconst struct reftable_log_record *lb = b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 00000000000..498e8c50bf4\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 00000000000..f4ad7cace41\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,412 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tint i;\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = {\n+\t\t\t\t.update = {\n+\t\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t\t.time = 1577123507,\n+\t\t\t\t\t.tz_offset = 100,\n+\t\t\t\t},\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].value.update.new_hash, 1);\n+\tset_test_hash(in[0].value.update.old_hash, 2);\n+\tfor (i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = {\n+\t\t\t\t.update = {\n+\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t\t},\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 00000000000..5370d2288c7\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t} value;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b..09d4b83ef9b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433405","messageId":"patch-v4-08.28-e2f2729443b-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 08/28] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:19Z","receivedAt":"2021-08-23T12:13:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be needed for reading reflog blocks in reftable.\n\nHelped-by: Carlo Marcelo Arenas Belón \nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 7 +++\n ci/lib.sh | 1 +\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n config.mak.uname | 1 +\n configure.ac | 13 ++++++\n 6 files changed, 115 insertions(+)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n\ndiff --git a/Makefile b/Makefile\nindex e98d8ed17cf..16c883978d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -256,6 +256,8 @@ all::\n #\n # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n #\n+# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n+#\n # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n #\n@@ -1738,6 +1740,11 @@ ifdef NO_DEFLATE_BOUND\n \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n endif\n \n+ifdef NO_UNCOMPRESS2\n+\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n+\tREFTABLE_OBJS += compat/zlib-uncompress2.o\n+endif\n+\n ifdef NO_POSIX_GOODIES\n \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n endif\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 476c3f369f5..5711c63979d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -224,6 +224,7 @@ linux-gcc-default)\n \t;;\n Linux32)\n \tCC=gcc\n+\tMAKEFLAGS=\"$MAKEFLAGS NO_UNCOMPRESS2=1\"\n \t;;\n linux-musl)\n \tCC=gcc\ndiff --git a/compat/.gitattributes b/compat/.gitattributes\nnew file mode 100644\nindex 00000000000..40dbfb170da\n--- /dev/null\n+++ b/compat/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\nnew file mode 100644\nindex 00000000000..6893bb469ce\n--- /dev/null\n+++ b/compat/zlib-uncompress2.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress2 (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 69413fb3dc0..61e11550b1f 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -256,6 +256,7 @@ ifeq ($(uname_S),FreeBSD)\n \tFILENO_IS_A_MACRO = UnfortunatelyYes\n endif\n ifeq ($(uname_S),OpenBSD)\n+\tNO_UNCOMPRESS2 = YesPlease\n \tNO_STRCASESTR = YesPlease\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\ndiff --git a/configure.ac b/configure.ac\nindex 031e8d3fee8..c3a913103d0 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n LIBS=\"$old_LIBS\"\n \n+AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n+AC_LANG_PROGRAM([#include ],\n+ [uncompress2(NULL,NULL,NULL,NULL);])])\n+AC_MSG_CHECKING([for uncompress2 in -lz])\n+old_LIBS=\"$LIBS\"\n+LIBS=\"$LIBS -lz\"\n+AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n+\t[AC_MSG_RESULT([yes])],\n+\t[AC_MSG_RESULT([no])\n+\tNO_UNCOMPRESS2=yes])\n+LIBS=\"$old_LIBS\"\n+\n GIT_UNSTASH_FLAGS($ZLIB_PATH)\n \n GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n+GIT_CONF_SUBST([NO_UNCOMPRESS2])\n \n #\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433406","messageId":"patch-v4-09.28-9d468fbdd07-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 09/28] reftable: reading/writing blocks","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:20Z","receivedAt":"2021-08-23T12:13:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/block.c | 448 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 120 +++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 698 insertions(+)\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 16c883978d4..1d396c87be9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2454,10 +2454,12 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 00000000000..eb5268dd3a6\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,448 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \n+\n+#ifdef NO_UNCOMPRESS2\n+/*\n+ * This is uncompress2, which is only available in zlib >= 1.2.9\n+ * (released as of early 2017)\n+ */\n+int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n+\t\tuLong *sourceLen);\n+#endif\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK !=\n+\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n+\t\t\t\tblock->data + block_header_skip, &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp && source.ops)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 00000000000..e207706a644\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 00000000000..4b3ea262dcb\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,120 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b..c9deeaf08c7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433407","messageId":"patch-v4-10.28-7d8b8e3b76a-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 10/28] reftable: a generic binary tree implementation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:21Z","receivedAt":"2021-08-23T12:13:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 1d396c87be9..be14b66f9e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,11 +2458,13 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 00000000000..82db7995dd6\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 00000000000..fbdd002e23a\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 00000000000..09a970e17b9\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = arg;\n+\tif (c->last) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7..050551fa698 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433408","messageId":"patch-v4-12.28-df4fae30ceb-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 12/28] reftable: generic interface to tables","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:23Z","receivedAt":"2021-08-23T12:13:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/generic.c | 169 +++++++++++++++++++++++++++++++++++\n reftable/generic.h | 32 +++++++\n reftable/reftable-generic.h | 47 ++++++++++\n reftable/reftable-iterator.h | 39 ++++++++\n reftable/reftable.c | 115 ++++++++++++++++++++++++\n 6 files changed, 405 insertions(+)\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex 9460f7de974..f45a981bc20 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,9 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n+REFTABLE_OBJS += reftable/generic.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/generic.c b/reftable/generic.c\nnew file mode 100644\nindex 00000000000..7a8a738d860\n--- /dev/null\n+++ b/reftable/generic.c\n@@ -0,0 +1,169 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = ~((uint64_t)0),\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_print(struct reftable_table *tab) {\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tuint32_t hash_id = reftable_table_hash_id(tab);\n+\tint err = reftable_table_seek_ref(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_table_seek_log(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn 0;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 00000000000..98886a06402\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 00000000000..d239751a778\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,47 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* dump table contents onto stdout for debugging */\n+int reftable_table_print(struct reftable_table *tab);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 00000000000..d3eee7af357\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 00000000000..0e4607a7cd6\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433409","messageId":"patch-v4-13.28-7e8a613206f-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 13/28] reftable: read reftable files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:24Z","receivedAt":"2021-08-23T12:13:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 194 +++++++++\n reftable/iter.h | 69 ++++\n reftable/reader.c | 801 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 +++\n reftable/reftable-reader.h | 101 +++++\n 6 files changed, 1233 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex f45a981bc20..5c857989091 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2456,7 +2456,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/generic.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 00000000000..93d04f735b8\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,194 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn !it->ops;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\treftable_free(it->offsets);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 00000000000..09eb0cbfa59\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,69 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+\n+/* Takes ownership of `offsets` */\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 00000000000..49f4ec070e6\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,801 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (!source->ops) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\tuint64_t file_size = block_source_size(source);\n+\n+\t/* Need +1 to read type of first block. */\n+\tuint32_t read_size = header_size(2) + 1; /* read v2 because it's larger. */\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\tif (read_size > file_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = block_source_read_block(source, &header, 0, read_size);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = file_size - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (!ti->bi.br) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next(ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek(tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id(tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+\n+int reftable_reader_print_file(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_reader *r = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\treftable_table_from_reader(&tab, r);\n+\terr = reftable_table_print(&tab);\n+done:\n+\treftable_reader_free(r);\n+\treturn err;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 00000000000..39583e5dbcd\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 00000000000..4a4bc2fdf85\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,101 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* print table onto stdout for debugging. */\n+int reftable_reader_print_file(const char *tablename);\n+\n+#endif\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433410","messageId":"patch-v4-14.28-ebe55aa6295-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 14/28] reftable: reftable file level tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:25Z","receivedAt":"2021-08-23T12:13:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/readwrite_test.c | 652 ++++++++++++++++++++++++++++++++++++++\n reftable/reftable-tests.h | 2 +-\n t/helper/test-reftable.c | 1 +\n 4 files changed, 655 insertions(+), 1 deletion(-)\n create mode 100644 reftable/readwrite_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5c857989091..5e2ff446894 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2469,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nnew file mode 100644\nindex 00000000000..5f6bcc2f775\n--- /dev/null\n+++ b/reftable/readwrite_test.c\n@@ -0,0 +1,652 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.value.update.new_hash = hash;\n+\t\tlog.value.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size(\n+\t\t\t\t(hash_id == GIT_SHA256_FORMAT_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record\n+\t\tlog = { .refname = \"refs/heads/master\",\n+\t\t\t.update_index = 0xa,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.value = { .update = {\n+\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t } } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.value.update.old_hash = hash1;\n+\tlog.value.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.value.update.old_hash = hash1;\n+\t\tlog.value.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n+\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_write_empty_table(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_write_key_order(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record refs[2] = {\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}, {\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}\n+\t};\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\terr = reftable_writer_add_ref(w, &refs[0]);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_add_ref(w, &refs[1]);\n+\tprintf(\"%d\\n\", err);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_writer_close(w);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_corrupt_table_empty(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+}\n+\n+static void test_corrupt_table(void)\n+{\n+\tuint8_t zeros[1024] = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err;\n+\tstrbuf_add(&buf, zeros, sizeof(zeros));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\tstrbuf_release(&buf);\n+}\n+\n+int readwrite_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_corrupt_table);\n+\tRUN_TEST(test_corrupt_table_empty);\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_write_key_order);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_write_empty_table);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654..3d541fa5c0c 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n-int reftable_test_main(int argc, const char **argv);\n+int readwrite_test_main(int argc, const char **argv);\n int stack_test_main(int argc, const char **argv);\n int tree_test_main(int argc, const char **argv);\n int reftable_dump_main(int argc, char *const *argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa698..898aba836fd 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433411","messageId":"patch-v4-15.28-a9880fc1a10-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 15/28] reftable: add a heap-based priority queue for reftable records","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:26Z","receivedAt":"2021-08-23T12:13:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 5e2ff446894..744a63b9d2c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n@@ -2468,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 00000000000..8918d158e2d\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 00000000000..385d2fb139a\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 00000000000..ad21673e854\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 3d541fa5c0c..0019cbcfa49 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ license that can be found in the LICENSE file or at\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int readwrite_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 898aba836fd..0b5a1701df1 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433412","messageId":"patch-v4-16.28-c073be56a6b-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 16/28] reftable: add merged table view","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:27Z","receivedAt":"2021-08-23T12:13:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 764 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 744a63b9d2c..defcc054fb6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2469,6 +2470,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 00000000000..e5b53da6db3\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,362 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (!mt) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record(tab, it, rec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 00000000000..8c4d4d58d77\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 00000000000..1e2afe37b8b\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,292 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n+\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == GIT_SHA1_FORMAT_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 00000000000..1a6d16915ab\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 0b5a1701df1..8087f2da4e6 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433413","messageId":"patch-v4-17.28-092b9715672-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 17/28] reftable: implement refname validation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:28Z","receivedAt":"2021-08-23T12:13:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 342 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex defcc054fb6..04b143f670d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2474,6 +2474,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 00000000000..95734969324\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 00000000000..a24b40fcb42\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 00000000000..8645cd93bbd\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 8087f2da4e6..c8db6852c35 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433414","messageId":"patch-v4-19.28-007c9359f42-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 19/28] reftable: add dump utility","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:30Z","receivedAt":"2021-08-23T12:13:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Carlo Marcelo Arenas Belón \n---\n reftable/dump.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 107 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 00000000000..155953d1b82\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,107 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"git-compat-util.h\"\n+#include \"hash.h\"\n+\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n+#include \"reftable-generic.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -6 sha256 hash format\\n\"\n+\t \" -h this help\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n+\t\t\tbreak;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n+\t\t\topt_dump_table = 1;\n+\t\telse if (!strcmp(\"-6\", argv[1]))\n+\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n+\t\t\topt_dump_stack = 1;\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n+\t\t\topt_compact = 1;\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t}\n+\n+\tif (argc != 2) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[1];\n+\n+\tif (opt_dump_table) {\n+\t\terr = reftable_reader_print_file(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = reftable_stack_print_directory(arg, opt_hash_id);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433415","messageId":"patch-v4-18.28-374c22efbd9-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 18/28] reftable: implement stack, a mutable database of reftable files.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:29Z","receivedAt":"2021-08-23T12:13:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-stack.h | 128 ++++\n reftable/stack.c | 1396 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 948 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2515 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 04b143f670d..152bbd6c134 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2475,6 +2475,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 00000000000..1b602dda58a\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. After calling this,\n+ * reftable_addition_destroy should still be called.\n+ */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+/* print the entire stack represented by the directory */\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 00000000000..48e22a6c184\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1396 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t/* On Windows, can only unlink after closing. */\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!rd) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i]) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t/* On Windows, can only unlink after closing. */\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (!names[i]) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len]) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (!add) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->time > 0 &&\n+\t\t log.value.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (!expiry && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (!dir) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_merged_table *merged = NULL;\n+\tstruct reftable_table table = { NULL };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\treftable_table_from_merged_table(&table, merged);\n+\terr = reftable_table_print(&table);\n+done:\n+\tif (stack)\n+\t\treftable_stack_destroy(stack);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 00000000000..f57005846e5\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 00000000000..890a5c01990\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,948 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static char *get_tmp_dir(const char *prefix)\n+{\n+\tchar *dir = get_tmp_template(prefix);\n+\tEXPECT(mkdtemp(dir));\n+\treturn dir;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i]; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\tprintf(\"testing print functionality:\\n\");\n+\terr = reftable_stack_print_directory(dir, GIT_SHA1_FORMAT_ID);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_print_directory(dir, GIT_SHA256_FORMAT_ID);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\n+\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .value = { .update = {\n+\t\t\t\t\t\t\t\t.new_hash = h1,\n+\t\t\t\t\t\t\t\t.old_hash = h2,\n+\t\t\t\t\t\t\t} } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.value.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.value.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.value.update.message, \"one\\n\"));\n+\n+\tinput.value.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.value.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\t/* even entries add the refs, odd entries delete them. */\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].value.update.new_hash =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t\t\tlogs[i].value.update.email =\n+\t\t\t\txstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].value.update.time = i;\n+\t\tlogs[i].value.update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].value.update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 100;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_reftable_stack_add);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c8db6852c35..996da85f7b5 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433416","messageId":"patch-v4-21.28-443bdebfb5d-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 21/28] git-prompt: prepare for reftable refs backend","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:32Z","receivedAt":"2021-08-23T12:13:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex db7c0068fb5..4177274bea4 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433418","messageId":"patch-v4-20.28-8f948c0b257-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 20/28] refs: RFC: Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:31Z","receivedAt":"2021-08-23T12:13:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see Documentation/technical/reftable.txt.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by setting GIT_TEST_REFTABLE in the environment. When\nGIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set.\n\nThere is no option to git-init for now, as the test suite still shows failures\nwith GIT_TEST_REFTABLE=1.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nHelped-by: Patrick Steinhardt \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 39 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/workdir/git-new-workdir | 2 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1691 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 8 +\n t/t0031-reftable.sh | 310 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 7 +-\n 23 files changed, 2159 insertions(+), 29 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdca..82c5940f143 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffd..72576235833 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex 152bbd6c134..6d0074939e5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -990,6 +990,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66fe66679c8..baa1ff4fc60 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c2f03f6018e..22b07d2b2fb 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -226,6 +228,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -235,6 +238,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -249,9 +270,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -268,7 +286,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -383,7 +401,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -422,6 +440,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -476,6 +495,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -549,6 +571,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -713,5 +736,5 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..d43d6898039 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n static int do_clear_stash(void)\n {\n \tstruct object_id obj;\n+\tint result;\n \tif (get_oid(ref_stash, &obj))\n \t\treturn 0;\n \n-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n+\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n+\n+\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n+\t * even when refs are deleted. */\n+\tdelete_reflog(ref_stash);\n+\treturn result;\n }\n \n static int clear_stash(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 0d0a80da61f..6f4facebc61 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -328,9 +329,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(null_oid()));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex 1e838303654..02d525ab9e0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -656,9 +656,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1076,6 +1077,7 @@ struct repository_format {\n \tint hash_algo;\n \tint sparse_index;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 61e11550b1f..019c88d5df4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -712,7 +712,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba1..1a25789d285 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a5215..989197aace0 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/refs.c b/refs.c\nindex e3b6d8f8dc0..4345c176a42 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,15 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1895,13 +1900,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1917,7 +1922,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1973,7 +1982,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1987,6 +1996,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -2000,9 +2010,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex a5685c891a9..c83ae18c692 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 7a3a61ac22f..8ec1047a0e0 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -676,6 +676,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..d972910bf21\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1691 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * two separate databases (ie. two reftable/ directories), one for the\n+ * repository, and one for the worktree. For reading, we merge the view (see\n+ * git_reftable_iterator) of both, when necessary.\n+ *\n+ * Unfortunately, the worktrees can also be selected by specifying a magic\n+ * refname (eg. worktree/BLA/refname, even if BLA isn't the current worktree.)\n+ */\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tconst char *wtname = refname;\n+\tint wtname_len = 0;\n+\tconst char *wtref = refname;\n+\n+\tif (refname == NULL)\n+\t\treturn store->main_stack;\n+\n+\tif (!parse_worktree_ref(refname, &wtname, &wtname_len, &wtref) &&\n+\t wtname_len) {\n+\t\t/* this makes me cry. Woe you if you try to access\n+\t\t * worktree/BLA/REF and the current worktree\n+\t\t * from the same process.\n+\t\t */\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_write_options cfg = {\n+\t\t\t.block_size = 4096,\n+\t\t\t.hash_id = the_hash_algo->format_id,\n+\t\t};\n+\n+\t\tstrbuf_addstr(&wt_dir, store->base.gitdir);\n+\t\tstrbuf_addstr(&wt_dir, \"/worktrees/\");\n+\t\tstrbuf_add(&wt_dir, wtname, wtname_len);\n+\t\tstrbuf_addstr(&wt_dir, \"/reftable\");\n+\n+\t\tif (store->worktree_stack)\n+\t\t\treftable_stack_destroy(store->worktree_stack);\n+\t\tstore->err = reftable_new_stack(&store->worktree_stack,\n+\t\t\t\t\t\twt_dir.buf, cfg);\n+\t\tassert(store->err != REFTABLE_API_ERROR);\n+\n+\t\treturn store->worktree_stack;\n+\t}\n+\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static const char *bare_ref_name(const char *ref)\n+{\n+\tconst char *out = ref;\n+\tint name_len = 0;\n+\tif (skip_prefix(ref, \"main-worktree/\", &out))\n+\t\treturn out;\n+\n+\tif (!parse_worktree_ref(ref, NULL, &name_len, &out) && name_len) {\n+\t\treturn out;\n+\t}\n+\n+\treturn ref;\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tlog->value.update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path. Mutates its arg, trimming it to the\n+ * gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf gitdir = STRBUF_INIT;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_realpath(&wt_buf, path, /*die_on_error=*/0);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tstrbuf_addbuf(&gitdir, &wt_buf);\n+\t} else {\n+\t\tstrbuf_realpath(&gitdir, path, /*die_on_error=*/0);\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir.buf);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir.buf);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\n+\t\trefs->err =\n+\t\t\treftable_new_stack(&refs->worktree_stack, sb.buf, cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\tstrbuf_release(&gitdir);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs,\n+\t\ttransaction->nr ? transaction->updates[0]->refname : NULL);\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_id = *null_oid();\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\tlog->value.update.message = u->msg;\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\terr = 0;\n+\t\t}\n+\n+\t\t/* XXX if this is a symref (say, HEAD), should we deref the\n+\t\t * symref and check the update.old_hash against the referent? */\n+\t\tif (ref.value_type == REFTABLE_REF_VAL2 ||\n+\t\t ref.value_type == REFTABLE_REF_VAL1)\n+\t\t\toidread(&old_id, ref.value.val1);\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\n+\t\tlog->value.update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.value.update.message = xstrdup(arg->logmsg);\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.value.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.value.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(\n+\t\trefs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.value.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.value.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.value.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.value.update.old_hash != NULL ||\n+\t\t log.value.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.value.update.message = NULL;\n+\t\tlog.value.update.old_hash = NULL;\n+\t\tlog.value.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\told_ref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].value.update.message = (char *)arg->logmsg;\n+\t\ttodo[0].value.update.old_hash = val1;\n+\t\ttodo[0].value.update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].value.update.old_hash = NULL;\n+\t\ttodo[1].value.update.new_hash = val1;\n+\t\ttodo[1].value.update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* this copies the entire reflog history. Is this the right semantics?\n+\t */\n+\t/* XXX should clear out existing reflog entries for oldname? */\n+\terr = reftable_merged_table_seek_log(\n+\t\treftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\toidread(&ri->oid, ri->log.value.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.value.update.name,\n+\t\t\t\t\t log.value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log.value.update.time, log.value.update.tz_offset,\n+\t\t\t log.value.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\trefname = bare_ref_name(refname);\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->value.update.old_hash);\n+\t\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->value.update.name,\n+\t\t\t\t\t log->value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t\t log->value.update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\trefname = bare_ref_name(refname);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len - 1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].value.update.old_hash);\n+\t\toidread(&noid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) &&\n+\t\t\t last_hash != NULL) {\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* XXX - skip writing records that were not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t} else {\n+\t\t/* XXX - print something */\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\trefname = bare_ref_name(refname); /* XXX - in which other cases should\n+\t\t\t\t\t we do this? */\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"Got reftable_ref_record with value type %d\",\n+\t\t ref.value_type);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex b2bf44c6faf..f1cc8df47c4 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -180,6 +180,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex 3740c93bc0f..1bd9b4d09c8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -82,6 +82,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex eb9367ca5cb..58504dfeb68 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -498,6 +498,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -648,6 +651,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1312,6 +1316,8 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\tthe_repository->repository_format_partial_clone =\n \t\t\t\trepo_fmt.partial_clone;\n \t\t\trepo_fmt.partial_clone = NULL;\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n \t\t}\n \t}\n \t/*\n@@ -1399,6 +1405,8 @@ void check_repository_format(struct repository_format *fmt)\n \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tthe_repository->repository_format_partial_clone =\n \t\txstrdup_or_null(fmt->partial_clone);\n+\t/* XXX why is repo->ref_storage_format set in multiple places?! */\n+\tthe_repository->ref_storage_format = xstrdup_or_null(fmt->ref_storage);\n \tclear_repository_format(&repo_fmt);\n }\n \ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 00000000000..d2b398ba5dc\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,310 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+write_script fake_editor <<\\EOF\n+echo \"$MSG\" >\"$1\"\n+echo \"$MSG\" >&2\n+EOF\n+GIT_EDITOR=./fake_editor\n+export GIT_EDITOR\n+\n+\n+test_expect_success 'using reftable' '\n+\tinitialize &&\n+\ttest -d .git/reftable &&\n+\ttest -f .git/reftable/tables.list\n+'\n+\n+test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n+\tinitialize &&\n+\ttest_commit 1st &&\n+\ttest_commit 2nd &&\n+\tMSG=b4 git notes add &&\n+\tMSG=b3 git notes edit &&\n+\techo b4 >expect &&\n+\tgit notes --ref commits@{1} show >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git reflog delete' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_commit file2 &&\n+\ttest_commit file3 &&\n+\ttest_commit file4 &&\n+\tgit reflog delete HEAD@{1} &&\n+\tgit reflog > output &&\n+\t! grep file3 output\n+'\n+\n+test_expect_success 'branch -D delete nonexistent branch' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_must_fail git branch -D ../../my-private-file\n+'\n+\n+test_expect_success 'branch copy' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\ttest_commit file2 &&\n+\tgit branch src &&\n+\tgit reflog src > expect &&\n+\tgit branch -c src dst &&\n+\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'update-ref on corrupted data' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tOLD_SHA1=$(git rev-parse HEAD) &&\n+\ttest_commit file2 &&\n+\tls -l .git/reftable &&\n+\tfor f in .git/reftable/*.ref\n+\tdo\n+\t\t>$f\n+\tdone &&\n+\ttest_must_fail git update-ref refs/heads/main $OLD_SHA1\n+'\n+\n+test_expect_success 'git stash' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttouch actual expected &&\n+\tgit -c status.showStash=true status >expected &&\n+\techo hoi >> file.t &&\n+\tgit stash push -m stashed &&\n+\tgit stash clear &&\n+\tgit -c status.showStash=true status >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'rename branch' '\n+\tinitialize &&\n+\tgit symbolic-ref HEAD refs/heads/before &&\n+\ttest_commit file &&\n+\tgit show-ref | sed s/before/after/g > expected &&\n+\tgit branch -M after &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\t(GIT_TEST_REFTABLE=1 git_init) &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(GIT_TEST_REFTABLE=1 git_init || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\t(GIT_TEST_REFTABLE=1 git_init start) &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tGIT_TEST_REFTABLE=1 git init -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat << EOF >expect &&\n+HEAD\n+refs/heads/main\n+refs/tags/m1\n+EOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506..cdc21bf2dcb 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; need files backend'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..6bdd430dfe3 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 577f32dc71f..e523c3dd624 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq !REFFILES\n+then\n+ skip_all='skipping pack-refs tests; requires files ref backend'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex abcfbed6d61..375e1dfd663 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1524,7 +1524,12 @@ parisc* | hppa*)\n \t;;\n esac\n \n-test_set_prereq REFFILES\n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq !REFFILES\n+else\n+ test_set_prereq REFFILES\n+fi\n \n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433417","messageId":"patch-v4-22.28-f76e7a7bd10-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 22/28] Add \"test-tool dump-reftable\" command.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:33Z","receivedAt":"2021-08-23T12:13:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0031-reftable.sh | 6 ++++++\n 5 files changed, 14 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 6d0074939e5..98b8f1a1f9f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2471,6 +2471,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 996da85f7b5..26b03d7b789 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex f7c888ffda7..338a57b104d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -61,6 +61,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 25f77469146..48cee1f4a2d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nindex d2b398ba5dc..c76e9042e1d 100755\n--- a/t/t0031-reftable.sh\n+++ b/t/t0031-reftable.sh\n@@ -288,6 +288,12 @@ test_expect_success 'FETCH_HEAD' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'dump reftable' '\n+\tinitialize &&\n+\thash_id=$(git config extensions.objectformat) &&\n+\ttest-tool dump-reftable $(test \"${hash_id}\" = \"sha256\" && echo \"-6\") -s .git/reftable\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433419","messageId":"patch-v4-23.28-6b62a8c52ae-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 23/28] t1301: document what needs to be done for reftable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:34Z","receivedAt":"2021-08-23T12:13:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1301-shared-repo.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh\nindex 84bf1970d8b..a5755b4a434 100755\n--- a/t/t1301-shared-repo.sh\n+++ b/t/t1301-shared-repo.sh\n@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' '\n \t)\n '\n \n+# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/\n for u in 002 022\n do\n-\ttest_expect_success POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n+\ttest_expect_success REFFILES,POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n \t\tmkdir sub && (\n \t\t\tcd sub &&\n \t\t\tumask $u &&\n@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' '\n+# For reftable, the check on .git/reftable/ is sufficient.\n+test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' '\n \tumask 077 &&\n \tgit config core.sharedRepository group &&\n \tgit reflog expire --all &&\n@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)'\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'template can set core.sharedrepository' '\n+test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' '\n \trm -rf child.git &&\n \tumask 0022 &&\n \tgit config core.sharedrepository 0666 &&\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433420","messageId":"patch-v4-26.28-dde80a982c1-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 26/28] reftable: fixup for new base topic 1/3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:37Z","receivedAt":"2021-08-23T12:13:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Lifted from Han-Wen's [1], with the addition of the removal of the\nreftable_error_to_errno() which is now unused. I also changed the:\n\n if err < 0\n if err > 1\n\nto just:\n\n if err < 0\n else if err\n\nWhich is clearer in this context, i.e. the reader doesn't need to\nsquint to see that the two if's are related, but different only in \"<\"\nv.s. \">\".\n\n1. https://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n refs/reftable-backend.c | 38 ++++++++------------------------------\n 1 file changed, 8 insertions(+), 30 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d972910bf21..dcc792e5e87 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \t\t\t\t const char *refname, struct object_id *oid,\n \t\t\t\t struct strbuf *referent,\n-\t\t\t\t unsigned int *type);\n+\t\t\t\t unsigned int *type, int *failure_errno);\n \n static void clear_reftable_log_record(struct reftable_log_record *log)\n {\n@@ -424,6 +424,7 @@ static int fixup_symrefs(struct ref_store *ref_store,\n \tstruct strbuf referent = STRBUF_INIT;\n \tint i = 0;\n \tint err = 0;\n+\tint failure_errno;\n \n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *update = transaction->updates[i];\n@@ -433,8 +434,8 @@ static int fixup_symrefs(struct ref_store *ref_store,\n \t\t\t\t\t\t&old_oid, &referent,\n \t\t\t\t\t\t/* mutate input, like\n \t\t\t\t\t\t files-backend.c */\n-\t\t\t\t\t\t&update->type);\n-\t\tif (err < 0 && errno == ENOENT &&\n+\t\t\t\t\t\t&update->type, &failure_errno);\n+\t\tif (err < 0 && failure_errno == ENOENT &&\n \t\t is_null_oid(&update->old_oid)) {\n \t\t\terr = 0;\n \t\t}\n@@ -1587,30 +1588,10 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn err;\n }\n \n-static int reftable_error_to_errno(int err)\n-{\n-\tswitch (err) {\n-\tcase REFTABLE_IO_ERROR:\n-\t\treturn EIO;\n-\tcase REFTABLE_FORMAT_ERROR:\n-\t\treturn EFAULT;\n-\tcase REFTABLE_NOT_EXIST_ERROR:\n-\t\treturn ENOENT;\n-\tcase REFTABLE_LOCK_ERROR:\n-\t\treturn EBUSY;\n-\tcase REFTABLE_API_ERROR:\n-\t\treturn EINVAL;\n-\tcase REFTABLE_ZLIB_ERROR:\n-\t\treturn EDOM;\n-\tdefault:\n-\t\treturn ERANGE;\n-\t}\n-}\n-\n static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \t\t\t\t const char *refname, struct object_id *oid,\n \t\t\t\t struct strbuf *referent,\n-\t\t\t\t unsigned int *type)\n+\t\t\t\t unsigned int *type, int *failure_errno)\n {\n \tstruct git_reftable_ref_store *refs =\n \t\t(struct git_reftable_ref_store *)ref_store;\n@@ -1633,13 +1614,10 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n \t}\n \n \terr = reftable_stack_read_ref(stack, refname, &ref);\n-\tif (err > 0) {\n-\t\terrno = ENOENT;\n-\t\terr = -1;\n-\t\tgoto done;\n-\t}\n \tif (err < 0) {\n-\t\terrno = reftable_error_to_errno(err);\n+\t\tgoto done;\n+\t} else if (err) {\n+\t\t*failure_errno = ENOENT;\n \t\terr = -1;\n \t\tgoto done;\n \t}\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433421","messageId":"patch-v4-25.28-0b3db9fc2d6-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 25/28] t1404: annotate test cases with REFFILES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:36Z","receivedAt":"2021-08-23T12:13:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\n* Reftable for now lacks detailed error messages for directory/file conflicts.\n Skip message comparisons.\n\n* Mark tests that muck with .git directly as REFFILES.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1404-update-ref-errors.sh | 56 +++++++++++++++++++++++++++---------\n 1 file changed, 42 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t1404-update-ref-errors.sh b/t/t1404-update-ref-errors.sh\nindex b729c1f4803..811d5bb56d4 100755\n--- a/t/t1404-update-ref-errors.sh\n+++ b/t/t1404-update-ref-errors.sh\n@@ -27,7 +27,9 @@ test_update_rejected () {\n \tfi &&\n \tprintf \"create $prefix/%s $C\\n\" $create >input &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_i18ngrep -F \"$error\" output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_i18ngrep -F \"$error\" output.err\n+\tfi &&\n \tgit for-each-ref $prefix >actual &&\n \ttest_cmp unchanged actual\n }\n@@ -101,7 +103,9 @@ df_test() {\n \t\tprintf \"%s\\n\" \"delete $delname\" \"create $addname $D\"\n \tfi >commands &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_cmp expected-err output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected-err output.err\n+\tfi &&\n \tprintf \"%s\\n\" \"$C $delref\" >expected-refs &&\n \tgit for-each-ref --format=\"%(objectname) %(refname)\" $prefix/r >actual-refs &&\n \ttest_cmp expected-refs actual-refs\n@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks update' '\n@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks create' '\n@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/foo $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks delete' '\n@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/foo $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect update' '\n@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect update' '\n@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect create' '\n@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect delete' '\n@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect no-deref update' '\n@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref update' '\n@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect no-deref create' '\n@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n@@ -463,7 +489,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success REFFILES 'non-empty directory blocks create' '\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433422","messageId":"patch-v4-24.28-c926f1526aa-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 24/28] t1401,t2011: parameterize HEAD.lock for REFFILES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:35Z","receivedAt":"2021-08-23T12:13:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1401-symbolic-ref.sh | 11 +++++++++--\n t/t2011-checkout-invalid-head.sh | 11 +++++++++--\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh\nindex 132a1b885ac..1b51013aded 100755\n--- a/t/t1401-symbolic-ref.sh\n+++ b/t/t1401-symbolic-ref.sh\n@@ -102,9 +102,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' '\n \ttest_cmp expect actual\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'symbolic-ref reports failure in exit code' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git symbolic-ref HEAD refs/heads/whatever\n '\n \ndiff --git a/t/t2011-checkout-invalid-head.sh b/t/t2011-checkout-invalid-head.sh\nindex e52022e1522..a56f7af442c 100755\n--- a/t/t2011-checkout-invalid-head.sh\n+++ b/t/t2011-checkout-invalid-head.sh\n@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' '\n \tgit checkout main --\n '\n \n+if test_have_prereq REFFILES\n+then\n+\tHEAD_LOCK=HEAD.lock\n+else\n+\tHEAD_LOCK=reftable/tables.list.lock\n+fi\n+\n test_expect_success 'checkout notices failure to lock HEAD' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git checkout -b other\n '\n \n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433423","messageId":"patch-v4-27.28-c4f9fb42d9e-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 27/28] reftable: fixup for new base topic 2/3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:38Z","receivedAt":"2021-08-23T12:14:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since my \"refs API: remove OID argument to reflog_expire()\" we don't\nhave the \"oid\" as part of the reflog_expire() signature. Instead the\nreflog_expire() should pass the OID of the tip of the \"locked\" ref to\nthe prepare_fn().\n\nIn files_reflog_expire() we do that by getting the OID from\nlock_ref_oid_basic(). I'm assuming (but am not familiar enough with\nreftable...) that by the time we get here we've got a locked ref\nalready in some way, so let's just use\nrefs_resolve_ref_unsafe_with_errno() to lookup the current OID of that\npresumably-locked ref.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n refs/reftable-backend.c | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex dcc792e5e87..94917c85cf7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1463,7 +1463,7 @@ static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n \n static int\n git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n-\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t unsigned int flags,\n \t\t\t reflog_expiry_prepare_fn prepare_fn,\n \t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n \t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n@@ -1497,6 +1497,9 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n \tstruct reftable_iterator it = { NULL };\n \tstruct reftable_addition *add = NULL;\n \tint err = 0;\n+\tint ignore_errno;\n+\tstruct object_id oid;\n+\n \tif (refs->err < 0) {\n \t\treturn refs->err;\n \t}\n@@ -1515,7 +1518,14 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n \tif (err) {\n \t\tgoto done;\n \t}\n-\tprepare_fn(refname, oid, policy_cb_data);\n+\tif (!refs_resolve_ref_unsafe_with_errno(ref_store, refname,\n+\t\t\t\t\t RESOLVE_REF_READING, &oid,\n+\t\t\t\t\t NULL, &ignore_errno)) {\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\n \twhile (1) {\n \t\tstruct reftable_log_record log = { NULL };\n \t\tint err = reftable_iterator_next_log(&it, &log);\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433424","messageId":"patch-v4-28.28-c863b03201c-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 28/28] reftable: fixup for new base topic 3/3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:39Z","receivedAt":"2021-08-23T12:14:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since the \"refs: add failure_errno to refs_read_raw_ref() signature\"\nin the base topic we've preferred to use\nrefs_resolve_ref_unsafe_with_errno() over the\nrefs_resolve_ref_unsafe().\n\nLet's make it explicitly clear that we're ignoring \"errno\" in these\ncases, and while we're at it change \"if (x != NULL)\" to just \"if (x)\"\nas refs_resolve_ref_unsafe*() returns a const char *, and explicitly\nchecking 0 or NULL in if's is not in line with our coding guidelines.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n refs/reftable-backend.c | 26 +++++++++++++++++---------\n 1 file changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 94917c85cf7..61ee144e19e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -315,9 +315,11 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tbreak;\n \t\tcase REFTABLE_REF_SYMREF: {\n \t\t\tint out_flags = 0;\n-\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\tint ignore_errno;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe_with_errno(\n \t\t\t\tri->ref_store, ri->ref.refname,\n-\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags,\n+\t\t\t\t&ignore_errno);\n \t\t\tri->base.flags = out_flags;\n \t\t\tif (resolved == NULL &&\n \t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n@@ -543,8 +545,10 @@ static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n {\n \tstruct object_id out_oid;\n \tint out_flags = 0;\n-\tconst char *resolved = refs_resolve_ref_unsafe(\n-\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tint ignore_errno;\n+\tconst char *resolved = refs_resolve_ref_unsafe_with_errno(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags,\n+\t\t&ignore_errno);\n \tif (is_null_oid(want_oid) != (resolved == NULL)) {\n \t\treturn REFTABLE_LOCK_ERROR;\n \t}\n@@ -845,20 +849,24 @@ static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n \t\tstruct reftable_log_record log = { NULL };\n \t\tstruct object_id new_oid;\n \t\tstruct object_id old_oid;\n+\t\tint ignore_errno;\n \n \t\tfill_reftable_log_record(&log);\n \t\tlog.refname = (char *)create->refname;\n \t\tlog.update_index = ts;\n \t\tlog.value.update.message = (char *)create->logmsg;\n-\t\tif (refs_resolve_ref_unsafe(\n+\t\tif (refs_resolve_ref_unsafe_with_errno(\n \t\t\t (struct ref_store *)create->refs, create->refname,\n-\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL,\n+\t\t\t &ignore_errno)) {\n \t\t\tlog.value.update.old_hash = old_oid.hash;\n \t\t}\n \n-\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n-\t\t\t\t\t create->target, RESOLVE_REF_READING,\n-\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\tif (refs_resolve_ref_unsafe_with_errno((struct ref_store *)create->refs,\n+\t\t\t\t\t\t create->target,\n+\t\t\t\t\t\t RESOLVE_REF_READING,\n+\t\t\t\t\t\t &new_oid, NULL,\n+\t\t\t\t\t\t &ignore_errno)) {\n \t\t\tlog.value.update.new_hash = new_oid.hash;\n \t\t}\n \n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433425","messageId":"patch-v4-11.28-7ec422318eb-20210823T120208Z-avarab@gmail.com","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"[PATCH v4 11/28] reftable: write reftable files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T12:12:22Z","receivedAt":"2021-08-23T12:15:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 148 ++++++++\n reftable/writer.c | 690 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex be14b66f9e3..9460f7de974 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2459,6 +2459,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 00000000000..af36462ced5\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \"reftable-record.h\"\n+\n+#include \n+#include /* ssize_t */\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 00000000000..3ca721e9f64\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,690 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\t\t\t 1 :\n+\t\t\t 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy(dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = -1;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->value.update.message;\n+\tif (!w->opts.exact_log_message && log->value.update.message) {\n+\t\tstrbuf_addstr(&cleaned_message, log->value.update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t/* multiple lines not allowed. */\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->value.update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->value.update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tif (arg->last) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 00000000000..09b88673d97\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tssize_t (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \n2.33.0.662.gbaddc25a55e\n\n"},{"id":"433814","messageId":"xmqqfsuwha5p.fsf@gitster.g","threadId":"56133","inReplyTo":"cover-v4-00.28-00000000000-20210823T120208Z-avarab@gmail.com","subject":"Re: [PATCH v4 00/28] Support reftable ref backend for Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-26T08:39:46Z","receivedAt":"2021-08-26T08:39:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This is a version of the reftable series queued on top of my\n> just-re-rolled fixes to the refs APIs, which it can make use of. For\n> the base topics see:\n>\n> https://lore.kernel.org/git/cover-v5-00.13-00000000000-20210823T113115Z-avarab@gmail.com/\n> https://lore.kernel.org/git/cover-v10-0.8-00000000000-20210823T114712Z-avarab@gmail.com/\n>\n> For Han-Wen's v3 of this see:\n> https://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\n>\n> I've got no desire to take over the reftable topic in its entirety,\n> but think given the rationale in\n> https://lore.kernel.org/git/877dgch4rn.fsf@evledraar.gmail.com/\n> (summarized in\n> https://lore.kernel.org/git/87y28sfokk.fsf@evledraar.gmail.com/) that\n> having the refs API fixes I noted above wait on the still-unstable\n> reftable doesn't make sense.\n\nOf course, you and Han-Wen are in much better position to judge the\nrelative merit to decide which one should go first than I am, but I\nhad an impression that the errno thing was even less stable, with\nAPI churn that deliberately broke the other topic in flight, which\nappeared to be just irresponsible.\n\n> I'll let Han-Wen deal with that squashing in a presumed future v5 of\n> this, assuming of course that Junio's happy with the plan of basing\n> hn/reftable on the refs API fixes above.\n>\n> I'm not sure that the fix I have in 27/28 is the right one, perhaps\n> we've already got information about what the tip OID of the refname is\n> at that point in git_reftable_reflog_expire() via some API I missed,\n> but that fix works, and is clearly more correct than the outstanding\n> segfault.\n"},{"id":"433815","messageId":"CAFQ2z_NBmKrVMeuTb=qeG_VSBWC7287-gGB9GhpokLZrmNiC0A@mail.gmail.com","threadId":"56133","inReplyTo":"xmqqfsuwha5p.fsf@gitster.g","subject":"Re: [PATCH v4 00/28] Support reftable ref backend for Git","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-26T08:56:28Z","receivedAt":"2021-08-26T08:56:42Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Thu, Aug 26, 2021 at 10:39 AM Junio C Hamano wrote:\n> > For Han-Wen's v3 of this see:\n> > https://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\n> >\n> > I've got no desire to take over the reftable topic in its entirety,\n> > but think given the rationale in\n> > https://lore.kernel.org/git/877dgch4rn.fsf@evledraar.gmail.com/\n> > (summarized in\n> > https://lore.kernel.org/git/87y28sfokk.fsf@evledraar.gmail.com/) that\n> > having the refs API fixes I noted above wait on the still-unstable\n> > reftable doesn't make sense.\n>\n> Of course, you and Han-Wen are in much better position to judge the\n> relative merit to decide which one should go first than I am, but I\n> had an impression that the errno thing was even less stable, with\n> API churn that deliberately broke the other topic in flight, which\n> appeared to be just irresponsible.\n\nThe bottom part of the errno series that I contributed has had ample\nscrutiny. It's a cleanup, and all-in-all much less experimental than\nthe reftable work. However, because it changes a calling convention\nin the ref backend API, it causes difficulty with other topics\n(notably: reftable). I would be in favor of graduating the series upto\n\"refs: make errno output explicit for read_raw_ref_fn\" early to\nprovide a stable basis for other patches.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"433839","messageId":"xmqqbl5kgsav.fsf@gitster.g","threadId":"56133","inReplyTo":"CAFQ2z_NBmKrVMeuTb=qeG_VSBWC7287-gGB9GhpokLZrmNiC0A@mail.gmail.com","subject":"Re: [PATCH v4 00/28] Support reftable ref backend for Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-26T15:05:28Z","receivedAt":"2021-08-26T15:05:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n> The bottom part of the errno series that I contributed has had ample\n> scrutiny. It's a cleanup, and all-in-all much less experimental than\n> the reftable work. However, because it changes a calling convention\n> in the ref backend API, it causes difficulty with other topics\n> (notably: reftable). I would be in favor of graduating the series upto\n> \"refs: make errno output explicit for read_raw_ref_fn\" early to\n> provide a stable basis for other patches.\n\nVery glad to see that the two of you are in agreement of the order\nand the approach. Let me replace the topics that have been queued\non 'seen' with the latest ones from Ævar, and we can go from there.\n\nThanks for a quick response. \n\n"},{"id":"433843","messageId":"87h7fcyyt2.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"877dgch4rn.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 00/25] Support reftable ref backend for Git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-26T16:02:49Z","receivedAt":"2021-08-26T16:07:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 23 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Tue, Aug 17 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n>\n>> This continues the work in https://github.com/git/git/pull/847, which the\n>> gitgitgadget erroneously closed.\n>>\n>> Changes relative to last series (version 20 Jul 2021, tip: 4a5891f )\n>>\n>> * fix HTTP serving\n>> * fix t1501 (use absolute paths.)\n>> * fix git-new-workdir\n>> * fixes for FreeBSD (thanks, carenas@gmail.com!)\n>>\n>\n> What's the overall status of this topic vis-a-vis whether it's being\n> proposed to be integrated or is in more RFC form?\n>\n> Since we discussed the non-working GIT_TEST_REFTABLE=true in [1] & [2]\n> there's been fixes to the REFFILES for the tests which has already\n> landed as c9780bb2ca8 (Merge branch 'hn/prep-tests-for-reftable',\n> 2021-07-13).\n>\n> But e.g. running t1410-reflog.sh reveals an assert failure() in this\n> library's own code:\n>\n> git: reftable/record.c:1075: reftable_record_copy_from: Assertion\n> `src->ops->type == rec->ops->type' failed.\n>\n> (A *lot* of things in the test suite still fail, I didn't look in any\n> detail. Just highlighting that it's not just remaining failures due to\n> hardcoding of .git/refs/*).\n>\n> I took your reply in [2] to mean that you were on board with the plan I\n> proposed in [1], but aside from the limited REFFILES fixes the structure\n> of what's here seems relatively unchanged.\n>\n> So is this series still considered RFC-esque, or what's the plan to move\n> this forward at this point?\n>\n> 1. https://lore.kernel.org/git/87h7jqz7k5.fsf@evledraar.gmail.com\n> 2. https://lore.kernel.org/git/CAFQ2z_P8vgY0RRT+XSH9K3VDQt39FLqXx6qfeZqaZPkwhq1w+A@mail.gmail.com\n\nJust a note: I didn't notice the outstanding REFFILES fixes that were\nabout to hit \"master\", so this was written with a branch rebased without\nthose fixes.\n\nBut in either case there's a lot of failures. Running the full test\nsuite also reveals at least a couple of tests running into this:\n\n BUG: refs.c:1039: free called on a prepared reference transaction\n\nt1400-update-ref.sh is one of them, t5510-fetch.sh is the other one. We\nthen have t5304-prune.sh, t3200-branch.sh and t2400-worktree-add.sh\nrunning into the assertion I noted above, in addition to the\nt1410-reflog.sh test I noted above.\n"},{"id":"433901","messageId":"xmqqwno7bgeo.fsf@gitster.g","threadId":"56133","inReplyTo":"patch-v4-18.28-374c22efbd9-20210823T120208Z-avarab@gmail.com","subject":"Re: [PATCH v4 18/28] reftable: implement stack, a mutable database of reftable files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-27T05:33:35Z","receivedAt":"2021-08-27T05:33:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This step does not compile under -pedantic because it assumes that\n__FUNCTION__ is available unconditionally (unlike trace.h where it\nallows __FUNCTION__ to be used iff compiler supports it).\n\nHere is a workaround that needs to be split and squashed into the\nsteps that introduce these test source files.\n\nSubject: [PATCH] SQUASH???\n https://github.com/git/git/runs/3439941236?check_suite_focus=true#step:5:700\n\n---\n reftable/basics_test.c | 10 ++---\n reftable/block_test.c | 2 +-\n reftable/merged_test.c | 6 +--\n reftable/pq_test.c | 2 +-\n reftable/readwrite_test.c | 30 +++++++--------\n reftable/record_test.c | 18 ++++-----\n reftable/refname_test.c | 2 +-\n reftable/stack_test.c | 80 +++++++++++++++++++--------------------\n reftable/test_framework.h | 2 +-\n reftable/tree_test.c | 2 +-\n 10 files changed, 77 insertions(+), 77 deletions(-)\n\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nindex 1fcd229725..ac1aed8a76 100644\n--- a/reftable/basics_test.c\n+++ b/reftable/basics_test.c\n@@ -24,7 +24,7 @@ static int binsearch_func(size_t i, void *void_args)\n \treturn args->key < args->arr[i];\n }\n \n-static void test_binsearch(void)\n+static void test_binsearch(const char *fn_name)\n {\n \tint arr[] = { 2, 4, 6, 8, 10 };\n \tsize_t sz = ARRAY_SIZE(arr);\n@@ -49,13 +49,13 @@ static void test_binsearch(void)\n \t}\n }\n \n-static void test_names_length(void)\n+static void test_names_length(const char *fn_name)\n {\n \tchar *a[] = { \"a\", \"b\", NULL };\n \tEXPECT(names_length(a) == 2);\n }\n \n-static void test_parse_names_normal(void)\n+static void test_parse_names_normal(const char *fn_name)\n {\n \tchar in[] = \"a\\nb\\n\";\n \tchar **out = NULL;\n@@ -66,7 +66,7 @@ static void test_parse_names_normal(void)\n \tfree_names(out);\n }\n \n-static void test_parse_names_drop_empty(void)\n+static void test_parse_names_drop_empty(const char *fn_name)\n {\n \tchar in[] = \"a\\n\\n\";\n \tchar **out = NULL;\n@@ -76,7 +76,7 @@ static void test_parse_names_drop_empty(void)\n \tfree_names(out);\n }\n \n-static void test_common_prefix(void)\n+static void test_common_prefix(const char *fn_name)\n {\n \tstruct strbuf s1 = STRBUF_INIT;\n \tstruct strbuf s2 = STRBUF_INIT;\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nindex 4b3ea262dc..8e21189f8d 100644\n--- a/reftable/block_test.c\n+++ b/reftable/block_test.c\n@@ -16,7 +16,7 @@ license that can be found in the LICENSE file or at\n #include \"test_framework.h\"\n #include \"reftable-tests.h\"\n \n-static void test_block_read_write(void)\n+static void test_block_read_write(const char *fn_name)\n {\n \tconst int header_off = 21; /* random */\n \tchar *names[30];\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 1e2afe37b8..f58e44003e 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -95,7 +95,7 @@ static void readers_destroy(struct reftable_reader **readers, size_t n)\n \treftable_free(readers);\n }\n \n-static void test_merged_between(void)\n+static void test_merged_between(const char *fn_name)\n {\n \tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n \n@@ -137,7 +137,7 @@ static void test_merged_between(void)\n \treftable_free(bs);\n }\n \n-static void test_merged(void)\n+static void test_merged(const char *fn_name)\n {\n \tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n \tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n@@ -237,7 +237,7 @@ static void test_merged(void)\n \treftable_free(bs);\n }\n \n-static void test_default_write_opts(void)\n+static void test_default_write_opts(const char *fn_name)\n {\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nindex ad21673e85..3537def65c 100644\n--- a/reftable/pq_test.c\n+++ b/reftable/pq_test.c\n@@ -15,7 +15,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable-tests.h\"\n #include \"test_framework.h\"\n \n-static void test_pq(void)\n+static void test_pq(const char *fn_name)\n {\n \tchar *names[54] = { NULL };\n \tint N = ARRAY_SIZE(names) - 1;\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex 5f6bcc2f77..5534f7fb67 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -20,7 +20,7 @@ license that can be found in the LICENSE file or at\n \n static const int update_index = 5;\n \n-static void test_buffer(void)\n+static void test_buffer(const char *fn_name)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_block_source source = { NULL };\n@@ -114,7 +114,7 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \treftable_writer_free(w);\n }\n \n-static void test_log_buffer_size(void)\n+static void test_log_buffer_size(const char *fn_name)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_write_options opts = {\n@@ -155,7 +155,7 @@ static void test_log_buffer_size(void)\n \tstrbuf_release(&buf);\n }\n \n-static void test_log_write_read(void)\n+static void test_log_write_read(const char *fn_name)\n {\n \tint N = 2;\n \tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n@@ -254,7 +254,7 @@ static void test_log_write_read(void)\n \treader_close(&rd);\n }\n \n-static void test_table_read_write_sequential(void)\n+static void test_table_read_write_sequential(const char *fn_name)\n {\n \tchar **names;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -296,7 +296,7 @@ static void test_table_read_write_sequential(void)\n \treader_close(&rd);\n }\n \n-static void test_table_write_small_table(void)\n+static void test_table_write_small_table(const char *fn_name)\n {\n \tchar **names;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -307,7 +307,7 @@ static void test_table_write_small_table(void)\n \tfree_names(names);\n }\n \n-static void test_table_read_api(void)\n+static void test_table_read_api(const char *fn_name)\n {\n \tchar **names;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -406,17 +406,17 @@ static void test_table_read_write_seek(int index, int hash_id)\n \treader_close(&rd);\n }\n \n-static void test_table_read_write_seek_linear(void)\n+static void test_table_read_write_seek_linear(const char *fn_name)\n {\n \ttest_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n }\n \n-static void test_table_read_write_seek_linear_sha256(void)\n+static void test_table_read_write_seek_linear_sha256(const char *fn_name)\n {\n \ttest_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n }\n \n-static void test_table_read_write_seek_index(void)\n+static void test_table_read_write_seek_index(const char *fn_name)\n {\n \ttest_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n }\n@@ -522,17 +522,17 @@ static void test_table_refs_for(int indexed)\n \treader_close(&rd);\n }\n \n-static void test_table_refs_for_no_index(void)\n+static void test_table_refs_for_no_index(const char *fn_name)\n {\n \ttest_table_refs_for(0);\n }\n \n-static void test_table_refs_for_obj_index(void)\n+static void test_table_refs_for_obj_index(const char *fn_name)\n {\n \ttest_table_refs_for(1);\n }\n \n-static void test_write_empty_table(void)\n+static void test_write_empty_table(const char *fn_name)\n {\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -568,7 +568,7 @@ static void test_write_empty_table(void)\n \tstrbuf_release(&buf);\n }\n \n-static void test_write_key_order(void)\n+static void test_write_key_order(const char *fn_name)\n {\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -604,7 +604,7 @@ static void test_write_key_order(void)\n \tstrbuf_release(&buf);\n }\n \n-static void test_corrupt_table_empty(void)\n+static void test_corrupt_table_empty(const char *fn_name)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_block_source source = { NULL };\n@@ -616,7 +616,7 @@ static void test_corrupt_table_empty(void)\n \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n }\n \n-static void test_corrupt_table(void)\n+static void test_corrupt_table(const char *fn_name)\n {\n \tuint8_t zeros[1024] = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex f4ad7cace4..f88a72b154 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -36,7 +36,7 @@ static void test_copy(struct reftable_record *rec)\n \treftable_record_destroy(©);\n }\n \n-static void test_varint_roundtrip(void)\n+static void test_varint_roundtrip(const char *fn_name)\n {\n \tuint64_t inputs[] = { 0,\n \t\t\t 1,\n@@ -68,7 +68,7 @@ static void test_varint_roundtrip(void)\n \t}\n }\n \n-static void test_common_prefix(void)\n+static void test_common_prefix(const char *fn_name)\n {\n \tstruct {\n \t\tconst char *a, *b;\n@@ -101,7 +101,7 @@ static void set_hash(uint8_t *h, int j)\n \t}\n }\n \n-static void test_reftable_ref_record_roundtrip(void)\n+static void test_reftable_ref_record_roundtrip(const char *fn_name)\n {\n \tint i = 0;\n \n@@ -164,7 +164,7 @@ static void test_reftable_ref_record_roundtrip(void)\n \t}\n }\n \n-static void test_reftable_log_record_equal(void)\n+static void test_reftable_log_record_equal(const char *fn_name)\n {\n \tstruct reftable_log_record in[2] = {\n \t\t{\n@@ -184,7 +184,7 @@ static void test_reftable_log_record_equal(void)\n \treftable_log_record_release(&in[1]);\n }\n \n-static void test_reftable_log_record_roundtrip(void)\n+static void test_reftable_log_record_roundtrip(const char *fn_name)\n {\n \tint i;\n \tstruct reftable_log_record in[2] = {\n@@ -258,7 +258,7 @@ static void test_reftable_log_record_roundtrip(void)\n \t}\n }\n \n-static void test_u24_roundtrip(void)\n+static void test_u24_roundtrip(const char *fn_name)\n {\n \tuint32_t in = 0x112233;\n \tuint8_t dest[3];\n@@ -268,7 +268,7 @@ static void test_u24_roundtrip(void)\n \tEXPECT(in == out);\n }\n \n-static void test_key_roundtrip(void)\n+static void test_key_roundtrip(const char *fn_name)\n {\n \tuint8_t buffer[1024] = { 0 };\n \tstruct string_view dest = {\n@@ -300,7 +300,7 @@ static void test_key_roundtrip(void)\n \tstrbuf_release(&roundtrip);\n }\n \n-static void test_reftable_obj_record_roundtrip(void)\n+static void test_reftable_obj_record_roundtrip(const char *fn_name)\n {\n \tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n \tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n@@ -358,7 +358,7 @@ static void test_reftable_obj_record_roundtrip(void)\n \t}\n }\n \n-static void test_reftable_index_record_roundtrip(void)\n+static void test_reftable_index_record_roundtrip(const char *fn_name)\n {\n \tstruct reftable_index_record in = {\n \t\t.offset = 42,\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nindex 8645cd93bb..81772179f9 100644\n--- a/reftable/refname_test.c\n+++ b/reftable/refname_test.c\n@@ -26,7 +26,7 @@ struct testcase {\n \tint error_code;\n };\n \n-static void test_conflict(void)\n+static void test_conflict(const char *fn_name)\n {\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 890a5c0199..c90b3a4bd9 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -62,9 +62,9 @@ static char *get_tmp_dir(const char *prefix)\n \treturn dir;\n }\n \n-static void test_read_file(void)\n+static void test_read_file(const char *fn_name)\n {\n-\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tchar *fn = get_tmp_template(fn_name);\n \tint fd = mkstemp(fn);\n \tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n \tint n, err;\n@@ -88,7 +88,7 @@ static void test_read_file(void)\n \tremove(fn);\n }\n \n-static void test_parse_names(void)\n+static void test_parse_names(const char *fn_name)\n {\n \tchar buf[] = \"line\\n\";\n \tchar **names = NULL;\n@@ -100,7 +100,7 @@ static void test_parse_names(void)\n \tfree_names(names);\n }\n \n-static void test_names_equal(void)\n+static void test_names_equal(const char *fn_name)\n {\n \tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n \tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n@@ -131,9 +131,9 @@ static int write_test_log(struct reftable_writer *wr, void *arg)\n \treturn reftable_writer_add_log(wr, wla->log);\n }\n \n-static void test_reftable_stack_add_one(void)\n+static void test_reftable_stack_add_one(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -169,12 +169,12 @@ static void test_reftable_stack_add_one(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_uptodate(void)\n+static void test_reftable_stack_uptodate(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL;\n \tstruct reftable_stack *st2 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tint err;\n \tstruct reftable_ref_record ref1 = {\n@@ -216,9 +216,9 @@ static void test_reftable_stack_uptodate(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_transaction_api(void)\n+static void test_reftable_stack_transaction_api(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -260,12 +260,12 @@ static void test_reftable_stack_transaction_api(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_validate_refname(void)\n+static void test_reftable_stack_validate_refname(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tint i;\n \tstruct reftable_ref_record ref = {\n@@ -303,9 +303,9 @@ static int write_error(struct reftable_writer *wr, void *arg)\n \treturn *((int *)arg);\n }\n \n-static void test_reftable_stack_update_index_check(void)\n+static void test_reftable_stack_update_index_check(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -335,9 +335,9 @@ static void test_reftable_stack_update_index_check(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_lock_failure(void)\n+static void test_reftable_stack_lock_failure(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -354,7 +354,7 @@ static void test_reftable_stack_lock_failure(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_add(void)\n+static void test_reftable_stack_add(const char *fn_name)\n {\n \tint i = 0;\n \tint err = 0;\n@@ -362,7 +362,7 @@ static void test_reftable_stack_add(void)\n \t\t.exact_log_message = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_ref_record refs[2] = { { NULL } };\n \tstruct reftable_log_record logs[2] = { { NULL } };\n@@ -436,14 +436,14 @@ static void test_reftable_stack_add(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_log_normalize(void)\n+static void test_reftable_stack_log_normalize(const char *fn_name)\n {\n \tint err = 0;\n \tstruct reftable_write_options cfg = {\n \t\t0,\n \t};\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \n \tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n@@ -492,10 +492,10 @@ static void test_reftable_stack_log_normalize(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_tombstone(void)\n+static void test_reftable_stack_tombstone(const char *fn_name)\n {\n \tint i = 0;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -575,9 +575,9 @@ static void test_reftable_stack_tombstone(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_hash_id(void)\n+static void test_reftable_stack_hash_id(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -619,14 +619,14 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n+static void test_log2(const char *fn_name)\n {\n \tEXPECT(1 == fastlog2(3));\n \tEXPECT(2 == fastlog2(4));\n \tEXPECT(2 == fastlog2(5));\n }\n \n-static void test_sizes_to_segments(void)\n+static void test_sizes_to_segments(const char *fn_name)\n {\n \tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n \t/* .................0 1 2 3 4 5 */\n@@ -644,7 +644,7 @@ static void test_sizes_to_segments(void)\n \treftable_free(segs);\n }\n \n-static void test_sizes_to_segments_empty(void)\n+static void test_sizes_to_segments_empty(const char *fn_name)\n {\n \tint seglen = 0;\n \tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n@@ -652,7 +652,7 @@ static void test_sizes_to_segments_empty(void)\n \treftable_free(segs);\n }\n \n-static void test_sizes_to_segments_all_equal(void)\n+static void test_sizes_to_segments_all_equal(const char *fn_name)\n {\n \tuint64_t sizes[] = { 5, 5 };\n \n@@ -665,7 +665,7 @@ static void test_sizes_to_segments_all_equal(void)\n \treftable_free(segs);\n }\n \n-static void test_suggest_compaction_segment(void)\n+static void test_suggest_compaction_segment(const char *fn_name)\n {\n \tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n \t/* .................0 1 2 3 4 5 6 */\n@@ -675,7 +675,7 @@ static void test_suggest_compaction_segment(void)\n \tEXPECT(min.end == 7);\n }\n \n-static void test_suggest_compaction_segment_nothing(void)\n+static void test_suggest_compaction_segment_nothing(const char *fn_name)\n {\n \tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n \tstruct segment result =\n@@ -683,9 +683,9 @@ static void test_suggest_compaction_segment_nothing(void)\n \tEXPECT(result.start == result.end);\n }\n \n-static void test_reflog_expire(void)\n+static void test_reflog_expire(const char *fn_name)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -761,12 +761,12 @@ static int write_nothing(struct reftable_writer *wr, void *arg)\n \treturn 0;\n }\n \n-static void test_empty_add(void)\n+static void test_empty_add(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tstruct reftable_stack *st2 = NULL;\n \n@@ -784,11 +784,11 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n-static void test_reftable_stack_auto_compaction(void)\n+static void test_reftable_stack_auto_compaction(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tint err, i;\n \tint N = 100;\n@@ -819,11 +819,11 @@ static void test_reftable_stack_auto_compaction(void)\n \tclear_dir(dir);\n }\n \n-static void test_reftable_stack_compaction_concurrent(void)\n+static void test_reftable_stack_compaction_concurrent(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tint err, i;\n \tint N = 3;\n@@ -869,11 +869,11 @@ static void unclean_stack_close(struct reftable_stack *st)\n \tFREE_AND_NULL(st->readers);\n }\n \n-static void test_reftable_stack_compaction_concurrent_clean(void)\n+static void test_reftable_stack_compaction_concurrent_clean(const char *fn_name)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(fn_name);\n \n \tint err, i;\n \tint N = 3;\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nindex 774cb275bf..4a40a8780a 100644\n--- a/reftable/test_framework.h\n+++ b/reftable/test_framework.h\n@@ -42,7 +42,7 @@ license that can be found in the LICENSE file or at\n #define RUN_TEST(f) \\\n \tfprintf(stderr, \"running %s\\n\", #f); \\\n \tfflush(stderr); \\\n-\tf();\n+\tf(#f);\n \n void set_test_hash(uint8_t *p, int i);\n \ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nindex 09a970e17b..4883f0de65 100644\n--- a/reftable/tree_test.c\n+++ b/reftable/tree_test.c\n@@ -31,7 +31,7 @@ static void check_increasing(void *arg, void *key)\n \tc->last = key;\n }\n \n-static void test_tree(void)\n+static void test_tree(const char *fn_name)\n {\n \tstruct tree_node *root = NULL;\n \n-- \n2.33.0-205-g1b5b9d966b\n\n"},{"id":"433904","messageId":"patch-1.1-f7d9c8af0c-20210827T055608Z-avarab@gmail.com","threadId":"56133","inReplyTo":"xmqqwno7bgeo.fsf@gitster.g","subject":"[RFC PATCH] reftable: fixup for broken __FUNCTION__ use","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-27T06:01:21Z","receivedAt":"2021-08-27T06:01:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The use of the __FUNCTION__ macro is non-standard, in this case all\nwe're doing with it is generating a prettier unique filename based on\nthe name of the function that makes it, presumably to make ad-hoc\ndebugging easier.\n\nFor mkdtemp() and mkstemp() we don't need to pass anything like this\nin, since the \"XXXXXX\" part of the template will ensure that we get a\nunique filename, but to make finding what function created what\ntempfile easy let's just use __LINE__ here, it's not *as easy*, but at\nleast this one uses standard behavior.\n\nThis can be tested under DEVOPTS=pedantic, i.e. before this change\nwe'd emit errors like:\n\n reftable/stack_test.c: In function ‘test_read_file’:\n reftable/stack_test.c:67:30: error: ISO C does not support ‘__FUNCTION__’ predefined identifier [-Werror=pedantic]\n char *fn = get_tmp_template(__FUNCTION__);\n\nThe current tip of \"seen\" is broken as a result, see\nhttps://github.com/git/git/runs/3439941236\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nOn Thu, Aug 26 2021, Junio C Hamano wrote:\n\n> This step does not compile under -pedantic because it assumes that\n> __FUNCTION__ is available unconditionally (unlike trace.h where it\n> allows __FUNCTION__ to be used iff compiler supports it).\n>\n> Here is a workaround that needs to be split and squashed into the\n> steps that introduce these test source files.\n>\n> Subject: [PATCH] SQUASH???\n> https://github.com/git/git/runs/3439941236?check_suite_focus=true#step:5:700\n\nFirst, thanks for the re-re-arrangement of the errno+reftable\ntopics. It looks like the \"seen\" integration is good, except for this\nissue under -pedantic.\n\nI can confirm your fix works, for what it's worth I came up with this\nalternate approach that I was about to send before I saw your proposed\nfixup.\n\nIt's smaller because it punts on the whole notion of adding the\nfunction name to the filename, as argued above I think __LINE__ should\nbe sufficient here (and is probably already overkill). The only reason\nto add __FUNCTION__ or __LINE__ to the filename is presumably for\none-off ad-hoc debugging.\n\nI see you pushed out seen a few minutes ago as bce8679d69 with your\nproposed squash. Let's leave that in and not have the churn of\nre-replacing that fixup. I'm submitting this more for Han-Wen's\nconsideration at this point.\n\n reftable/stack_test.c | 43 ++++++++++++++++++++++---------------------\n 1 file changed, 22 insertions(+), 21 deletions(-)\n\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 890a5c0199..f6b542b259 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -46,25 +46,26 @@ static int count_dir_entries(const char *dirname)\n \treturn len;\n }\n \n-static char *get_tmp_template(const char *prefix)\n+static char *get_tmp_template(const int line)\n {\n \tconst char *tmp = getenv(\"TMPDIR\");\n \tstatic char template[1024];\n-\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n-\t\t tmp ? tmp : \"/tmp\", prefix);\n+\tsnprintf(template, sizeof(template) - 1, \"%s/stack-test.c-%d.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", line);\n+\tfprintf(stderr, \"have template %s\\n\", template);\n \treturn template;\n }\n \n-static char *get_tmp_dir(const char *prefix)\n+static char *get_tmp_dir(const int line)\n {\n-\tchar *dir = get_tmp_template(prefix);\n+\tchar *dir = get_tmp_template(line);\n \tEXPECT(mkdtemp(dir));\n \treturn dir;\n }\n \n static void test_read_file(void)\n {\n-\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tchar *fn = get_tmp_template(__LINE__);\n \tint fd = mkstemp(fn);\n \tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n \tint n, err;\n@@ -133,7 +134,7 @@ static int write_test_log(struct reftable_writer *wr, void *arg)\n \n static void test_reftable_stack_add_one(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -174,7 +175,7 @@ static void test_reftable_stack_uptodate(void)\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL;\n \tstruct reftable_stack *st2 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tint err;\n \tstruct reftable_ref_record ref1 = {\n@@ -218,7 +219,7 @@ static void test_reftable_stack_uptodate(void)\n \n static void test_reftable_stack_transaction_api(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -265,7 +266,7 @@ static void test_reftable_stack_validate_refname(void)\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tint i;\n \tstruct reftable_ref_record ref = {\n@@ -305,7 +306,7 @@ static int write_error(struct reftable_writer *wr, void *arg)\n \n static void test_reftable_stack_update_index_check(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -337,7 +338,7 @@ static void test_reftable_stack_update_index_check(void)\n \n static void test_reftable_stack_lock_failure(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -362,7 +363,7 @@ static void test_reftable_stack_add(void)\n \t\t.exact_log_message = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_ref_record refs[2] = { { NULL } };\n \tstruct reftable_log_record logs[2] = { { NULL } };\n@@ -443,7 +444,7 @@ static void test_reftable_stack_log_normalize(void)\n \t\t0,\n \t};\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \n \tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n@@ -495,7 +496,7 @@ static void test_reftable_stack_log_normalize(void)\n static void test_reftable_stack_tombstone(void)\n {\n \tint i = 0;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -577,7 +578,7 @@ static void test_reftable_stack_tombstone(void)\n \n static void test_reftable_stack_hash_id(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -685,7 +686,7 @@ static void test_suggest_compaction_segment_nothing(void)\n \n static void test_reflog_expire(void)\n {\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n@@ -766,7 +767,7 @@ static void test_empty_add(void)\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tstruct reftable_stack *st2 = NULL;\n \n@@ -788,7 +789,7 @@ static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tint err, i;\n \tint N = 100;\n@@ -823,7 +824,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tint err, i;\n \tint N = 3;\n@@ -873,7 +874,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n-\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\tchar *dir = get_tmp_dir(__LINE__);\n \n \tint err, i;\n \tint N = 3;\n-- \n2.33.0.736.g68690aaec9a\n\n"},{"id":"433907","messageId":"CAPUEspj4AbB+8O80vQ7etOwn3XMs3Gnc-cBc8=WaCDg6+zruQg@mail.gmail.com","threadId":"56133","inReplyTo":"patch-1.1-f7d9c8af0c-20210827T055608Z-avarab@gmail.com","subject":"Re: [RFC PATCH] reftable: fixup for broken __FUNCTION__ use","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-27T07:00:44Z","receivedAt":"2021-08-27T07:01:08Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Thu, Aug 26, 2021 at 11:01 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> The use of the __FUNCTION__ macro is non-standard\n\nand is also problematic because it is a \"magic\" variable[1] instead of\na string literal.\n\nFWIW, __func__ has also the same problem but it is part of C99; I\nstill like Junio's solution\nbetter than my own that was using those.\n\n[1] https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html\n"},{"id":"434067","messageId":"CAFQ2z_MRrxBgpP8wy32pAuzQb0BAEH-jSUZA01Atxs4DzRQvKQ@mail.gmail.com","threadId":"56133","inReplyTo":"patch-1.1-f7d9c8af0c-20210827T055608Z-avarab@gmail.com","subject":"Re: [RFC PATCH] reftable: fixup for broken __FUNCTION__ use","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-30T12:11:12Z","receivedAt":"2021-08-30T12:11:27Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Aug 27, 2021 at 8:01 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> The use of the __FUNCTION__ macro is non-standard, in this case all\n> we're doing with it is generating a prettier unique filename based on\n> the name of the function that makes it, presumably to make ad-hoc\n> debugging easier.\n..\n> It's smaller because it punts on the whole notion of adding the\n> function name to the filename, as argued above I think __LINE__ should\n> be sufficient here (and is probably already overkill). The only reason\n> to add __FUNCTION__ or __LINE__ to the filename is presumably for\n> one-off ad-hoc debugging.\n\nIt's there so it's easy to track down which tests forget to cleanup\nafter they run, and __LINE__ works for this as well.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"434069","messageId":"CAFQ2z_MnJr58Cu970Be7SXE1jFkgOGThQ0Geq2+kNihJE3_BvQ@mail.gmail.com","threadId":"56133","inReplyTo":"patch-v4-27.28-c4f9fb42d9e-20210823T120208Z-avarab@gmail.com","subject":"Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-30T12:32:10Z","receivedAt":"2021-08-30T12:32:25Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Aug 23, 2021 at 2:13 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> Since my \"refs API: remove OID argument to reflog_expire()\" we don't\n> have the \"oid\" as part of the reflog_expire() signature. Instead the\n> reflog_expire() should pass the OID of the tip of the \"locked\" ref to\n> the prepare_fn().\n>\n> In files_reflog_expire() we do that by getting the OID from\n> lock_ref_oid_basic(). I'm assuming (but am not familiar enough with\n> reftable...) that by the time we get here we've got a locked ref\n> already in some way, so let's just use\n> refs_resolve_ref_unsafe_with_errno() to lookup the current OID of that\n> presumably-locked ref.\n\nI quickly looked at the files code, but I don't understand why the OID\nneeds to be passed-in (before your refactoring): in builtin/reflog.c\n(before), the current OID is read, with any protection. This means\nthat its value can't be trusted.\n\nAfter your refactoring, you lock the ref. I guess in the files backend\nthis protects against non-atomic update of (ref, reflog) racing with a\nconcurrent reflog expiry? In reftable, the (ref,reflog) update is\natomic, so there is no need for locking to properly sequence\noperations.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"434072","messageId":"87r1ebxe2y.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"CAFQ2z_MnJr58Cu970Be7SXE1jFkgOGThQ0Geq2+kNihJE3_BvQ@mail.gmail.com","subject":"Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-30T13:01:50Z","receivedAt":"2021-08-30T13:22:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 30 2021, Han-Wen Nienhuys wrote:\n\n> On Mon, Aug 23, 2021 at 2:13 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> Since my \"refs API: remove OID argument to reflog_expire()\" we don't\n>> have the \"oid\" as part of the reflog_expire() signature. Instead the\n>> reflog_expire() should pass the OID of the tip of the \"locked\" ref to\n>> the prepare_fn().\n>>\n>> In files_reflog_expire() we do that by getting the OID from\n>> lock_ref_oid_basic(). I'm assuming (but am not familiar enough with\n>> reftable...) that by the time we get here we've got a locked ref\n>> already in some way, so let's just use\n>> refs_resolve_ref_unsafe_with_errno() to lookup the current OID of that\n>> presumably-locked ref.\n>\n> I quickly looked at the files code, but I don't understand why the OID\n> needs to be passed-in (before your refactoring): in builtin/reflog.c\n> (before), the current OID is read, with any protection. This means\n> that its value can't be trusted.\n>\n> After your refactoring, you lock the ref. I guess in the files backend\n> this protects against non-atomic update of (ref, reflog) racing with a\n> concurrent reflog expiry? In reftable, the (ref,reflog) update is\n> atomic, so there is no need for locking to properly sequence\n> operations.\n\nBefore my [1] we'd do:\n\n 1. Read the OID for the branch in builtin/reflog.c\n 2. Pass it to refs/files-backend.c, it would lock at that OID (or fail if it changed)\n 3. Pass the OID with the now-locked OID to the builtin/reflog.c code\n\nAfter that [1] we do:\n\n 1. Lock the branch in refs/files-backend.c\n 2. Pass the OID with the now-locked OID to the builtin/reflog.c code\n (ditto #3 above)\n\nWhatever reftable itself does with updates doesn't change that we need\nto do that 2nd step of passing the OID to builtin/reflog.c, as it makes\nuse of it.\n\nThat code is a bit confusing, if you want to understand it I recommend\nreading it at the tip of my yet-unsubmitted\navar/cleanup-refs-api-and-reflog-expire-post-no-eisdir, it makes the\ncontrol flow a lot cleaner.\n\nSo as far as what we do here is concerned, we're stuck with the refs\nfiles backend inherently wanting to pass \"I locked this for you, here's\nthe OID\".\n\nI guess it could also pass \"OK, now go ahead and expire\" and pass no\nOID. We'd then in builtin/reflog.c lookup the current OID for the logic\nthere, but just having the reftable backend appease the common API by\nlooking up the OID and passing it seemed like the most straightforward\nthing to do.\n\nI haven't tested this or thought it through, but I don't understand how\nreftable isn't going to race in reflog expiry then. Sure, the ref/reflog\nupdate itself is atomic, so it won't suffer from the needing-a-lock\nproblem of two concurrent file backend writers doing the equivalent of:\n\n echo $NEW_SHA1 >.git/refs/heads/some-branch\n\nBut we will need at least the optimistic locking of code like\nbuiltin/reflog.c wanting to do an expiry, and deciding whether to do\nthat expiry based on a given state of the ref/reflog. I.e. we don't\nwant:\n\n 1. Start reflog expiry\n 2. Code in builtin/reflog.c looks up the OID\n 3. Code in builtin/reflog.c decides whether expire the reflog\n 4. Concurrent with #4, another writer updates the ref/reflog pair\n 5. Code in builtin/reflog.c says \"OK, expire it!\"\n 6. Reftable queues a delete/prune of the reflog per #5.\n\nThis would be a sequente of updates to the ref/reflog, none of whom were\nracy as far as the reftable semantics itself are concerneb, but where\nwe'd do the wrong thing because the writer thought we had A when we\nreally had B. So we need the equivalent of an \"git update-ref\" with the\n\"\".\n\nIs there a better way to do that in this case that I'm missing?\n\n1. https://lore.kernel.org/git/patch-v5-09.13-aba12606cea-20210823T113115Z-avarab@gmail.com/\n2. https://github.com/avar/git/compare/avar/files-backend-remove-dead-errno-eisdir-6...avar:avar/cleanup-refs-api-and-reflog-expire-post-no-eisdir\n"},{"id":"434074","messageId":"CAFQ2z_OTdXSvPcjPRi8fx6F_yjwiL3vRpPtnEOTyaeBxMB2wHQ@mail.gmail.com","threadId":"56133","inReplyTo":"871r6kh3m8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 20/25] refs: RFC: Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-30T13:31:55Z","receivedAt":"2021-08-30T13:32:12Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Aug 23, 2021 at 12:12 PM Ævar Arnfjörð Bjarmason\n wrote:\n> Elsewhere in this series we use the ARRAY_SIZE() macro from\n> git-compat-util.h, can't we also use REALLOC_ARRAY() from the same\n> header here?\n\nDone.\n\n> > [...]\n> > + if (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n> > + /* XXX - skip writing records that were not changed. */\n> > + err = reftable_addition_commit(add);\n> > + } else {\n> > + /* XXX - print something */\n> > + }\n>\n> Aren't these comments covered by some of the failing tests under\n> GIT_TEST_REFTABLE=true? I.e. what I mentioned\n> https://lore.kernel.org/git/877dgch4rn.fsf@evledraar.gmail.com/\n\nThe XXX comments are directed at reviewers. I don't know if these\nplaces relate to test failures.\n\n> > [...]\n> > + if (err < 0) {\n> > + errno = reftable_error_to_errno(err);\n> > + err = -1;\n> > + goto done;\n> > + }\n>\n> In your proposed fixup for the merger of our topics in\n> https://lore.kernel.org/git/pull.1054.v3.git.git.1629207607.gitgitgadget@gmail.com/\n> you have the call to reftable_error_to_errno() here deleted, so isn't\n> this also redundant at this point (and then the\n> reftable_error_to_errno() function can be deleted), or is this errno\n> setting still needed with some of my changes?\n\nNo; it's purely informational, but the errno codes aren't a good\nmatch, so OK to drop as well.\n\n> Can't later tests just use \"main\" instead of primary with a\n> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main, then we can drop this\n> \"git_init\" wrapper?\n\nDone.\n\n> > +initialize () {\n> > + rm -rf .git &&\n>\n> Should instead set up a test_when_finished \"rm -rf .git\" ?\n\nNo. The test framework sets up a scratch repo, but the scratch repo\ndoesn't use reftable, so it interferes with the tests.\n\n> > + (GIT_TEST_REFTABLE=1; export GIT_TEST_REFTABLE; git_init) &&\n>\n> This export before calling git_init can surely go away if git_init goes,\n> but alo why export beforehand here, but in later tests in this file we\n> just do a plain:\n>\n> (GIT_TEST_REFTABLE=1 git_init) &&\n\nDone.\n\n> > + mv .git/hooks .git/hooks-disabled\n>\n> Is this \"mv\" cargo-culted from what test_create_repo() used to do before\n> my-f0d4d398e28 (test-lib: split up and deprecate test_create_repo(),\n> 2021-05-10)? In any case templated hooks are disabled by default (named\n> *.sample), so I don't think this is needed.\n\nDone.\n\n> > + echo hoi >> file.t &&\n>\n> Nit: >>file.t, not \">> file.t\".\n\nDnoe.\n\n> > + git show-ref | sed s/before/after/g > expected &&\n>\n> Don't have \"git\" on the LHS of a pipe, it'll hide a segfault. Should use\n> a temporary file. Also \"s/ > />/g\" like above (and in some places below,\n> will stop noting it...).\n\nDone.\n\n> > +test_expect_success 'SHA256 support, env' '\n> > + rm -rf .git &&\n> > + GIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n> > + (GIT_TEST_REFTABLE=1 git_init) &&\n> > + mv .git/hooks .git/hooks-disabled &&\n>\n> Comments about this .git/hooks-disabled not being needed aside, this\n> seems to be duplicating the initialize() function. I.e. should we not\n> skip the \"rm -rf\" and \"mv\" here, and just set GIT_DEFAULT_HASH=sha256\n> and call initialize?\n>\n> (Better yet, if we get rid of that \"git init\" wrapper as I noted above,\n> this can just be an argument to \"git init\", no?\n\nDone.\n\n>\n> > [...]\n> > + rm -rf .git &&\n> > + (GIT_TEST_REFTABLE=1 git_init --object-format=sha256) &&\n> > + mv .git/hooks .git/hooks-disabled &&\n>\n> ditto.\n\ndone.\n\n> > + ! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n>\n> Always \"test_must_fail git\", not \"! git\".\n\ndone.\n\n>\n> > [...]\n> > +test_expect_success 'clone calls transaction_initial_commit' '\n> > + test_commit message1 file1 &&\n> > + git clone . cloned &&\n> > + (test -f cloned/file1 || echo \"Fixme.\")\n>\n> So this test really tests nothing much, and we should skip the \"Fixme\"\n> here and have this be test_expect_failure() or something instead?\n>\n> > [...]\n> > + git show-ref | cut -f2 -d\" \" >actual &&\n>\n> Git on LHS of a pipe again.\n>\n> > +# This matches show-ref's output\n> > +print_ref() {\n> > + echo \"$(git rev-parse \"$1\") $1\"\n> > +}\n> > +\n> > +test_expect_success 'peeled tags are stored' '\n> > + initialize &&\n> > + test_commit file &&\n> > + git tag -m \"annotated tag\" test_tag HEAD &&\n> > + {\n> > + print_ref \"refs/heads/primary\" &&\n> > + print_ref \"refs/tags/file\" &&\n> > + print_ref \"refs/tags/test_tag\" &&\n> > + print_ref \"refs/tags/test_tag^{}\"\n> > + } >expect &&\n>\n> Maybe I'm missing something, but wouldn't this print_ref() helper be\n> better as a \"git for-each-ref --format\" of some sort?\n>\n> > [...]\n> > +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> > +start_httpd\n> > +\n> > +REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n>\n> Let's split these httpd-needing tests into another test file, see\n> https://lore.kernel.org/git/87bl753i2p.fsf@evledraar.gmail.com/ for why.\n\ndone.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"434078","messageId":"CAFQ2z_Pa1fG935rQBQPijE=YHatsHbsLd_f8dT7Zh-grdgwbjg@mail.gmail.com","threadId":"56133","inReplyTo":"87r1ebxe2y.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-08-30T13:48:48Z","receivedAt":"2021-08-30T13:49:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Aug 30, 2021 at 3:22 PM Ævar Arnfjörð Bjarmason\n wrote:\n> But we will need at least the optimistic locking of code like\n> builtin/reflog.c wanting to do an expiry, and deciding whether to do\n> that expiry based on a given state of the ref/reflog. I.e. we don't\n> want:\n>\n> 1. Start reflog expiry\n> 2. Code in builtin/reflog.c looks up the OID\n> 3. Code in builtin/reflog.c decides whether expire the reflog\n> 4. Concurrent with #4, another writer updates the ref/reflog pair\n> 5. Code in builtin/reflog.c says \"OK, expire it!\"\n> 6. Reftable queues a delete/prune of the reflog per #5.\n>\n> This would be a sequente of updates to the ref/reflog, none of whom were\n> racy as far as the reftable semantics itself are concerneb, but where\n> we'd do the wrong thing because the writer thought we had A when we\n> really had B. So we need the equivalent of an \"git update-ref\" with the\n> \"\".\n>\n> Is there a better way to do that in this case that I'm missing?\n\nI spent some more time looking at builtin/reflog.c, but I am still not\n100% sure what the locking is used for.\n\nFrom a quick glance, the OID goes into tip_commit, and tip_commit goes\ninto a reachable list (?). The reachable list is then for something,\nbut I can't really tell what.\n\nIn your example with 1.-6., it's still not clear to me what the\nundesired behavior is precisely. If the reflog is pruned in #6, is the\nworry that the update in #4 is pruned immediately after being\neffected?\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"434082","messageId":"87lf4jxbxn.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"CAFQ2z_Pa1fG935rQBQPijE=YHatsHbsLd_f8dT7Zh-grdgwbjg@mail.gmail.com","subject":"Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-30T14:03:42Z","receivedAt":"2021-08-30T14:08:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 30 2021, Han-Wen Nienhuys wrote:\n\n> On Mon, Aug 30, 2021 at 3:22 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>> But we will need at least the optimistic locking of code like\n>> builtin/reflog.c wanting to do an expiry, and deciding whether to do\n>> that expiry based on a given state of the ref/reflog. I.e. we don't\n>> want:\n>>\n>> 1. Start reflog expiry\n>> 2. Code in builtin/reflog.c looks up the OID\n>> 3. Code in builtin/reflog.c decides whether expire the reflog\n>> 4. Concurrent with #4, another writer updates the ref/reflog pair\n>> 5. Code in builtin/reflog.c says \"OK, expire it!\"\n>> 6. Reftable queues a delete/prune of the reflog per #5.\n>>\n>> This would be a sequente of updates to the ref/reflog, none of whom were\n>> racy as far as the reftable semantics itself are concerneb, but where\n>> we'd do the wrong thing because the writer thought we had A when we\n>> really had B. So we need the equivalent of an \"git update-ref\" with the\n>> \"\".\n>>\n>> Is there a better way to do that in this case that I'm missing?\n>\n> I spent some more time looking at builtin/reflog.c, but I am still not\n> 100% sure what the locking is used for.\n>\n> From a quick glance, the OID goes into tip_commit, and tip_commit goes\n> into a reachable list (?). The reachable list is then for something,\n> but I can't really tell what.\n>\n> In your example with 1.-6., it's still not clear to me what the\n> undesired behavior is precisely. If the reflog is pruned in #6, is the\n> worry that the update in #4 is pruned immediately after being\n> effected?\n\nYes, I think so. But I'm not sure. I skimmed the code quickly today, and\nwhen I wrote the referenced series didn't focus much on the nitty-gritty\nof the builtin/reflog.c behavior other than assuring myself that I was\ndoing the exact same thing as before as far as its logic was concerned.\n\nI.e. it always locked at a given OID. Before my in-flight \"reflog\nexpire: don't lock reflogs using previously seen OI\" it might not lock\nbut get this error:\n\n error: cannot lock ref '': ref '' is at but expected \n\nBut at least it wouldn't do anything, but the current code does require\nthe passed-in OID. See the code that needs \"unreachable_expire_kind\" and\n\"tip_commit\".\n\nPerhaps that whole thing can also be refactored somehow. If I change the\n\"commit = lookup_commit(the_repository, oid);\" in\n\"reflog_expiry_prepare()\" to just \"commit = NULL\" all tests pass, but\nthat might just be missing test coverage in the face of same race...\n"},{"id":"434083","messageId":"87ilznxbt0.fsf@evledraar.gmail.com","threadId":"56133","inReplyTo":"CAFQ2z_OTdXSvPcjPRi8fx6F_yjwiL3vRpPtnEOTyaeBxMB2wHQ@mail.gmail.com","subject":"Re: [PATCH v3 20/25] refs: RFC: Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-30T14:10:02Z","receivedAt":"2021-08-30T14:11:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 30 2021, Han-Wen Nienhuys wrote:\n\n> On Mon, Aug 23, 2021 at 12:12 PM Ævar Arnfjörð Bjarmason\n> [ omitting addressed points]\n>> > +initialize () {\n>> > + rm -rf .git &&\n>>\n>> Should instead set up a test_when_finished \"rm -rf .git\" ?\n>\n> No. The test framework sets up a scratch repo, but the scratch repo\n> doesn't use reftable, so it interferes with the tests.\n\nPerhaps \"TEST_NO_CREATE_REPO=1\" at the top of the test is more\nstraightforward in this case?\n"}]}